The keyword here is: "queuing". Lots of theory.

Suppose you have a router with 3 interfaces, A, B and C. An ethernetcard is an interface, or a E1 2 Mbps link, or an OC-192 10 Gbps link.
There are hundreds of types of interfaces. For routers they are all kindof the same, they transport bits between routers. Suppose interfaces A, B and C are all the same speed.

What happens when a packet arrives on interface B ?
The router will recieve the packet bit by bit. It will store it in a buffer. When the whole packet has been received, the router will check some stuff (crc, ttl), and look at the destination IP address.
Then look in the routing table. Suppose the destination is reachable via another router that is attached on interface A. The router will send the packet out on interface A. Simple.

Now what happens when 2 packets arrive on both interface B and interface C, almost at the same time ?
Cal them packet B1 and B2. Both are stored in a buffer. Suppose packet B1 is received a little earlier than C1, 1 millisecond.
When the last bit on B1 is received, the router looks at the destination IP address, and in our example starts sending it out on interface A.
You should realize that sending a packet takes time. A 1500 byte packet (12000 bits) on a 64 Kbps link takes 188 milliseconds to complete.
Now C1 is received on C while the B1 is being transmitted out on A,the router wants to send C1 out over A too,but interface A is busy,so C1 needs to wait until B1 has completley been transmitted.

While C1 waits, another packet might be received on interface C. Or even on interface B.
Those packets need to be stored in a buffer too. Or else they have to be thrown away.
Throwing away is allowed in the Internet,the source host originally sending the packet needs to retransmit the packet,but retransmitting is not always good.
Suppose a packet is send over a 10 hop path, and the router on the 8th hop keeps throwing away half of the packets.
That means many packets are being transmitted over many links for nothing and those retransmitted packets use transmission capacity on those links.
If routers would throw away packets lightly,many packets would need to be retransmitted, causing lots and lots of extra traffic.
And extra traffic causes more congestion, which is the root cause of dropping packets in the first place.

Therefor routers try to buffer packets.
In the case of our hypothetical router with interfaces A, B and C, the router will dedicate some of its memory as buffers. It might create 64 buffers per interface.
Now if there are bursts of traffic, the router can buffer some extra packets, and when the burst is over before all buffers are full, there will be no dropped packets, and the overall performance of the network will be better.

So how much buffers should a router have ?
Of course buffers are in RAM, and RAM costs money,so you can't have infinite buffers.
And even if you had huge buffers, then you don't want routers to buffer forever.
Suppose you send a ping packet through a congested network, and it takes 5 minutes before it comes back, that would be useless.
So you want some buffering, but not too much.
ISPs and router vendors have concluded that the best thing is to have 200 milliseconds worth of buffering on an interface.
E.g. if you have a 10 Mbps connection, then you want 200 KBytes of buffering on that interface.
Buffering on WAN is more important than LAN, because 1) LAN is usually higher, thus less congestion anyway, and 2) WAN links are more expensive, so you want as few retransmission as possible.
When those buffers are filled up, a router will drop extra packets.
Even if it had extra buffers it wouldn't help much anyway.
The original sender will notice the packet loss (it won't get an ack from its peer), and retransmit.
Noticing the packetloss will also cause TCP to slow down, which should help prevent the congestion.

The result of all of this is: the Internet is optimized for large datatransfers, FTP and HTTP.
Transfers will not notice congestion or short hiccups, Buffers on routers will keep the flow of packets smooth, without retransmissions.

But for games this is not too good.
When an interface is congested, we will notice our packets being stuck in the buffers for a while. 200 ms buffering at each congestion point will be very noticable in your UT ping. Both F1 and F6.
And if congestion becomes worse, we will also notice the router dropping our packets in our PacketLoss statistics.

How to solve these problems ?
Well, the IETF (Internet engineers) have not come up with a good standard solution yet.
Building a network with lots of bandwidth is always good, many ISPs do that.
Many backbones are only loaded 10% or 20% of capacity.
You will notice congestion not in the backbones, but either between backbones (peering points, exchanges), or in the last mile (your connection to your ISP).

Queuing.
Those buffers I was talking about are grouped in "queues", Each interface has an "input queue" and a "output queue".
When a router has finished sending a packet out interface X, it will check the output queue of interface X, and start sending the next packet.
In the simplest way the order of taking packet out of the queue is simple FIFO (first in, first out), But there are smarter queueing strategies.
Weighted Fair Queuing is one of them. Check Google for more info.
Basically the router will create multiple output queues per interface, and each stream of packets gets its own queue.
Then the router can pick packets from those queues in a fair way.
The result is, if someone does heavy FTP, and you do UT, you both will get the same bandwidth, even though the FTP would try to get all bandwidth.

You could ask your ISP to enable Fair Queuing on the interface of the router that is connected to your building.
Usually queing should be done on the ISP's router. The congestion is usually when traffic is incoming. Configuring WFQ on your PC or on your own router will not help then.

Anyway, long story. I'm sure you knew half of this already. I hope WFQ helps a bit. There is also something called RED that might help. Ask your ISP for WFQ/RED.