Avatar

How to save your halo.bungie.net data before Feb 9th. (Gaming)

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Sunday, January 31, 2021, 17:50 (1391 days ago)
edited by Joe Duplessie (SNIPE 316), Sunday, January 31, 2021, 18:17

With halo.bungie.net going down Feb 9th, there's been a scramble to save data. While community efforts are underway on that, I thought I'd share the best way I've found to save a copy of the Halo data for your Gamertag(s). I tried to make this idiot-proof, so keep that in mind as I over-explain things.

IMPORTANT TIPS:
-You likely won't be able to fit all your stats into one run of the program. That's fine, you can run it many times and combine the files later.
-I suggest you run the program multiple times to account for bungie.net errors. Between the different copies you should get all your data. You can combine the files later, but for now just make sure you have everything.
-This process can take a while. Don't wait! The deadline is fast approaching. I give many tips to speed things up.
-Don't run the program more than two times at once. That tends to make bungie.net give more errors. Slow and steady.
-Backup your files. I describe my preferred way at the end.

First download the .txt files I provide here. They'll make the process of entering the URLs much faster and easier. Use Notepad to read them.
Next install HTTrack.
Once that's installed, open it up, click "Next" and you'll see this screen: Name the project whatever you want. The format I use is "B.net GAMERTAG." Category doesn't matter. Put anything, or leave it blank. I put "b.net stats". After that, click "Next".

On this screen, we set the pages to download (1). There's a generous character limit, but if you're like me, you played more than enough to surpass it. If so, you have to split your stats into multiple projects (I suggest every game have its own file). We will set it to download every page of your game history, which links to every game you played. First thing to do is set the limits and exceptions. It isn't strictly necessary, but it significantly speeds up the download. Click the "Set options..." button (2).

An options box will pop up. Click the "Limits" tab, and you'll see this. Set the "Maximum mirroring depth" (1) and "Maximum external depth" (2) to "2". You can do more, but that will dramatically increase the size of your download, and how long it will take. Now click "Max transfer rate (B/s)" (3). Set it to blank, which will make it download as fast as Bungie will allow. Next, click over to the "Scan Rules" tab (4). Looks like this. Open the "B.net Exceptions and Campaign History Quick" text file, and copy/paste the first 21 addresses (making sure the "-" symbol is in front of them) into the text box on the "Scan Rules" tab in the options menu. Once you see them in the text box, click "OK" (2) to save and exit.

To find out what URLs we need, we have to go to halo.bungie.net. Once there, select the drop down menu (1), and click the game you want to find your stats in (2). Enter your gamertag in the search bar (3), then click the Marathon logo (4) to search.

For Halo 2
H2 is very simple. In the .txt file "B.net H2 History Quick" replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All)
Now find how many pages worth of H2 you played and copy/paste the correct amount of pages from that document into the URL list of your HTTrack project. If you played enough games to exceed the URL box's character limit, you have to split it into multiple projects.

For Halo 3 Matchmaking
H3 starts off simple enough with Matchmaking. Click the "Halo 3" tab on halo.bungie.net (1), then click the "Matchmaking History" tab (2) and note the amount of pages, as seen here. Now with that number in mind, load up the "B.net H3 MM History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from that document into the URL list of your HTTrack project (making sure to include the first 6 links as well). If you played enough games to exceed the URL box's character limit, you have to split it into multiple projects.

For Halo 3 Customs
First Click the "Halo 3" tab on halo.bungie.net (1), then click the "Custom Game History" tab (2). Here's where things get a little messy. The page count may be inaccurate until you find the last page. For example, it might say "83 pages", but if you click on page 83 you get an error. Then if you click on page 77, it says only 77 pages. So What you'll have to do is click through until you find the true last page. Once you've found it, open the "B.net H3 Customs History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from that document into the URL list of your HTTrack project. If you played enough games to exceed the URL box's character limit, you have to split it into multiple projects.

For Halo 3 Campaign
Back to simplicity. Click the "Campaign History" tab under "Halo 3" on halo.bungie.net and note the amount of pages. Now with that number in mind, load up the "B.net Exceptions and Campaign History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from the second section of that document into the URL list of your HTTrack project.

For ODST
To find how many pages you need to capture for ODST, on halo.bungie.net click the "Halo 3: ODST" tab. Under that select "Campaign History". Note how many pages there are to capture and open the "B.net Exceptions and Campaign History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from the third section of that document into the URL list of your HTTrack project.
Now for firefight it's the same process. on halo.bungie.net click the "Halo 3: ODST" tab. Under that select "Firefight History". Note how many pages there are to capture and open the "B.net Exceptions and Campaign History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from the fourth section of that document into the URL list of your HTTrack project.

For Reach
Reach can be hard to find out your pages. At your Halo: Reach service record on halo.bungie.net, note how many games it says you've played (1), this will give us a good idea of how many pages there will be. Now click "Game History" on the left (2). What you have to do is find the last page manually. To do this scroll all the way down and click "Next". You could just hit the "Next" button over and over, but it's much quicker to edit the number at the end of the URL. Every page contains 25 games, so take the total amount of games you played, and divide by 25. If you played 1000 games of Reach, you'll want to go to page 40. Or as in our example image, 1,487 games means go to page 59. It isn't always exact, so you may have to change the URL one by one to find the last page. Once you've found the last page, open the "B.net HR History Quick" .txt file and replace all instances of "GAMER+TAG" with your desired gamertag (in Notepad, click "Edit">"Replace...">"Find what: GAMER+TAG>Replace with: Your Tag>Replace All). Copy/paste the correct amount of pages from that document into the URL list of your HTTrack project (making sure to include the first 12 links as well). If you played enough games to exceed the URL box's character limit, you have to split it into multiple projects.

Once you've got all your pages in the URL box of your HTTrack project (or as many as you could fit), click "Next". On the next screen, click "Finish". The program will begin saving your pages. Depending on how much you played, this can take a long time. Hours. Also bungie.net gives errors fairly frequently, so I highly recommend you run the program multiple times to ensure you get all of your data.

Running it again
The second time you run the program will be far easier, because it will save your settings from the first time. But you have to be careful not to overwrite the first copy,
When it finishes the first time, click "Finish". Now from the title, click the down arrow and select the file you want to run again. Click "Next" ONLY ONCE. We have to go move some files around before proceeding.
no to wherever you saved your data to, (the default being This PC>Windows(C:)>C:\My Web Sites) and make a new folder (right click>New>Folder). Give the new folder a name similar to your saved data's. For example, if you named it "B.net GAMERTAG", then name the new folder "B.net GAMERTAG Complete".
Now back to HTTrack, click the "Action" drop down menu and change it from "Update existing download" to "Download website(s)". Click "Next", and then click "Finish".

Backing up your files
Once you have all your files, and however many copies you felt like doing, right click the containing folder and select Send to>Compressed (zipped) folder. That may take a while but once it's done, we're ready to upload it somewhere. There are many places to do this (and you should do as many as you can), but the one I'll focus on is Archive.org.
After creating an account and signing in, click the "UPLOAD" button in the top-right. Now click the green "Upload Files" button. Then click the blue "Choose files to upload" button. Find and select the zip folder you created, then you'll see this screen. Add a description (1), for example "All stats on halo.bungie.net for player GAMERTAG on Feb 1st, 2021.". Next you have to add some tags, separated by commas. I suggest "halo, bungie, bungie.net, stats, GAMERTAG". Thirdly, you'll have to select what collection this upload belongs to (3). Select "Community Data". Now just click "Upload and Create Your Item" (4), wait for the upload to finish and you're done!

Avatar

How to save your halo.bungie.net data before Feb 9th.

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Monday, February 01, 2021, 12:45 (1390 days ago) @ Joe Duplessie (SNIPE 316)

I should mention, Gamertags containing spaces (like mine) should be fine. However, if you're worried about the URLs not working you can replace the spaces with a "+" sign and it will work for sure.

Avatar

Thanks for posting this!

by Kermit @, Raleigh, NC, Tuesday, February 02, 2021, 07:20 (1389 days ago) @ Joe Duplessie (SNIPE 316)

You mentioned community efforts to save this stuff. Any worth mentioning?

Avatar

Thanks for posting this!

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Tuesday, February 02, 2021, 14:41 (1389 days ago) @ Kermit

You mentioned community efforts to save this stuff. Any worth mentioning?

https://www.reddit.com/r/halo/comments/l6jzpb/halobungienet_data_archive_almost_done_thank_you/

This is probably the biggest ongoing effort. As of writing they've gotten 95% of all H2 data, and 99% of H3. No word on ODST and Reach.

Also Archive.org team apparently got all H3 data in 2015 https://twitter.com/textfiles/status/1349924896286121987

No word there if they are trying to get the other games.

I really wish Bungie would just give the data to 343, but it's nice to see so many people coming together for this.

Do you know iif anyone is saving the news articles?

by Claude Errera @, Tuesday, February 02, 2021, 16:06 (1389 days ago) @ Joe Duplessie (SNIPE 316)

We've gotten a request to do so, I don't have the tools (or the time, really) to help.

Avatar

Do you know iif anyone is saving the news articles?

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Tuesday, February 02, 2021, 18:27 (1389 days ago) @ Claude Errera
edited by Joe Duplessie (SNIPE 316), Tuesday, February 02, 2021, 19:24

I thought bungie switched them over to b.net proper. If not then uh oh...

Doesn't HBO have a BWU archive? I'm guessing that doesn't cover everything.

Edit: Okay after looking at it, there are 258 pages. What I can do is download it, and Wayback machine each page. I'll upload the file to Archive.org, then it'll be there for whenever you get the time. I'll let you know when I'm done. :)

Avatar

Here it is.

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Wednesday, February 03, 2021, 05:16 (1388 days ago) @ Joe Duplessie (SNIPE 316)

https://archive.org/details/b.net-articles

With all the images and whatnot I expected it to take much longer, and be a much larger file. Pleasant surprise. I normally run the program twice to account for errors, but I did it 3 times with this one. Still working on putting each page on Wayback.

Avatar

Here it is.

by Cody Miller @, Music of the Spheres - Never Forgot, Wednesday, February 03, 2021, 09:47 (1388 days ago) @ Joe Duplessie (SNIPE 316)

https://archive.org/details/b.net-articles

With all the images and whatnot I expected it to take much longer, and be a much larger file. Pleasant surprise. I normally run the program twice to account for errors, but I did it 3 times with this one. Still working on putting each page on Wayback.

You are indeed the man. I'm sure there's lot of good history in there now preserved.

Avatar

Ok so what is .whtt

by Cody Miller @, Music of the Spheres - Never Forgot, Wednesday, February 03, 2021, 10:03 (1388 days ago) @ Joe Duplessie (SNIPE 316)

…and how do I read these articles?

Avatar

Ok so what is .whtt

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Wednesday, February 03, 2021, 11:17 (1388 days ago) @ Cody Miller

That's the program's configuration file. You don't need it.

All you need to do is open the index file. The pages should all link together from there.

There's 100 pages each in the first 2 files, and the remaining 57 in the last. Each of the 3 files has 2 backups, in case of errors.

This is fantastic. Thanks.

by Claude Errera @, Wednesday, February 03, 2021, 12:09 (1388 days ago) @ Joe Duplessie (SNIPE 316)

I don't know that I'll ever get to a place where this is mirrored, en masse... but the fact that it exists is spectacular, and means that I CAN put up individual articles if they're asked for (and I have the time).

Not lost to time = really, really good

Avatar

Ok so what is .whtt

by Cody Miller @, Music of the Spheres - Never Forgot, Wednesday, February 03, 2021, 12:12 (1388 days ago) @ Joe Duplessie (SNIPE 316)

That's the program's configuration file. You don't need it.

All you need to do is open the index file. The pages should all link together from there.

There's 100 pages each in the first 2 files, and the remaining 57 in the last. Each of the 3 files has 2 backups, in case of errors.

Oh okay I see now.

Thanks! This rules.

Avatar

Bravo!

by Kermit @, Raleigh, NC, Wednesday, February 03, 2021, 12:13 (1388 days ago) @ Joe Duplessie (SNIPE 316)

Way to go, Snipe!

Avatar

This is fantastic. Thanks.

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Wednesday, February 03, 2021, 13:11 (1388 days ago) @ Claude Errera

I don't know that I'll ever get to a place where this is mirrored, en masse... but the fact that it exists is spectacular, and means that I CAN put up individual articles if they're asked for (and I have the time).

Not lost to time = really, really good

Thanks for bringing it up, because I had definitely not thought to save it.

Avatar

Amazing Job, Man!

by Morpheus @, High Charity, Thursday, February 04, 2021, 18:12 (1387 days ago) @ Joe Duplessie (SNIPE 316)

- No text -

Avatar

Thanks!

by squidnh3, Sunday, February 07, 2021, 12:24 (1384 days ago) @ Joe Duplessie (SNIPE 316)

I at least got my Halo 2 stuff, but I'm getting a ton of error pages trying to get the Halo 3 stuff. Even just accessing the pages normally it's probably less than 50/50 that a give page of results loads at all. I wonder if there are too many people trying to do this at the moment.

Avatar

I'm having trouble with the H3 stuff as well.

by Kermit @, Raleigh, NC, Sunday, February 07, 2021, 14:36 (1384 days ago) @ squidnh3

- No text -

Avatar

Thanks!

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Sunday, February 07, 2021, 15:19 (1384 days ago) @ squidnh3

B.net is getting hit really hard right now. Multiple teams trying to get everything as fast as possible. There's not much you can do at this point, my apologies. Good luck.

Avatar

Thanks!

by squidnh3, Sunday, February 07, 2021, 16:51 (1384 days ago) @ Joe Duplessie (SNIPE 316)

I guess that's good anyway, hopefully it becomes accessible elsewhere. I'll give it another try tomorrow morning when maybe traffic will be lower.

Avatar

It worked

by squidnh3, Monday, February 08, 2021, 06:13 (1383 days ago) @ Joe Duplessie (SNIPE 316)
edited by squidnh3, Monday, February 08, 2021, 06:16

Was able to get the lists of all the games.

Do you know of any efficient way to get the stats of each individual game? Setting the mirror depth to 2 seemed to pick them up but also a lot of duplication that would likely take forever.

Thanks again! You still looking for Flawless Raider in D1?

Avatar

It worked

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Monday, February 08, 2021, 09:45 (1383 days ago) @ squidnh3

Was able to get the lists of all the games.

Do you know of any efficient way to get the stats of each individual game? Setting the mirror depth to 2 seemed to pick them up but also a lot of duplication that would likely take forever.

If the mirroring depth to 2 should have gotten them all. Though it's been discovered that things with drop- down menus don't seem to save properly.

Thanks again! You still looking for Flawless Raider in D1?

I sure am! Just not sure when is a good time to try with D2 getting updates and stuff.

Avatar

It worked

by squidnh3, Monday, February 08, 2021, 09:53 (1383 days ago) @ Joe Duplessie (SNIPE 316)

If the mirroring depth to 2 should have gotten them all. Though it's been discovered that things with drop- down menus don't seem to save properly.

Damn, I misread that part of your guide. Site seems overwhelmed again now, maybe I'll try again tonight. Not sure when on Feb 9th they plan to pull the plug.

I sure am! Just not sure when is a good time to try with D2 getting updates and stuff.

The new season starts tomorrow, but I'm sure in a week or so there will be some lulls.

Avatar

It worked

by Kermit @, Raleigh, NC, Monday, February 08, 2021, 09:57 (1383 days ago) @ squidnh3

If the mirroring depth to 2 should have gotten them all. Though it's been discovered that things with drop- down menus don't seem to save properly.


Damn, I misread that part of your guide. Site seems overwhelmed again now, maybe I'll try again tonight. Not sure when on Feb 9th they plan to pull the plug.

I sure am! Just not sure when is a good time to try with D2 getting updates and stuff.


The new season starts tomorrow, but I'm sure in a week or so there will be some lulls.

Even though it would probably doom us to failure, I can try to help. Crota is the only flawless I've ever done.

Avatar

It worked

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Monday, February 08, 2021, 12:07 (1383 days ago) @ squidnh3

The new season starts tomorrow, but I'm sure in a week or so there will be some lulls.

Well maybe I'll give it a shot sometime soon.

Avatar

Thanks for posting this!

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Monday, February 08, 2021, 14:06 (1383 days ago) @ Joe Duplessie (SNIPE 316)

https://twitter.com/textfiles/status/1357035759673442307?s=19

They're getting Reach too. It would seem like between everyone's efforts, everything has been saved!

Avatar

halo.bungie.net must be getting hammered.

by INSANEdrive, ಥ_ಥ | f(ಠ‿↼)z | ᕕ( ᐛ )ᕗ| ¯\_(ツ)_/¯, Monday, February 08, 2021, 18:18 (1383 days ago) @ Joe Duplessie (SNIPE 316)

Trying to grab my pictures I made back in the day, but I can't even get on. Keep seeing a "pardon our dust" errors. And even when I do manage to squeeze in, I get errors on the Highres DLs.

Oh Well. Was able to get one of my favorites at least.

[image]

Avatar

"It's dead, Jim" V_V | (RIP halo.bungie.net)

by INSANEdrive, ಥ_ಥ | f(ಠ‿↼)z | ᕕ( ᐛ )ᕗ| ¯\_(ツ)_/¯, Tuesday, February 09, 2021, 09:06 (1382 days ago) @ INSANEdrive
edited by INSANEdrive, Tuesday, February 09, 2021, 09:11

- No text -

Avatar

A few more fun things I saved from Bungie.net.

by Joe Duplessie (SNIPE 316) ⌂ @, Detroit, Wednesday, February 17, 2021, 16:05 (1374 days ago) @ Joe Duplessie (SNIPE 316)

First up, all of the taglines I could find for the front page. There may have been more, but I refreshed a LOT of times. (Click to see HUGE image).
[image]

Now some error art. These four showed up when a page failed to load a specific page's data.
[image] [image] [image] [image]

And searching through the internet, I found some old error art that they stopped using at some point.
[image]

This one appeared if the site failed to load a certain screenshot correctly.
[image]

And a similar one if it couldn't find the screenshot at all.
[image]

Here's one for if it couldn't find a player you searched for.
[image]

And finally, this showed up if it failed to load a page completely.
[image]

Avatar

Doing fine work

by Cody Miller @, Music of the Spheres - Never Forgot, Wednesday, February 17, 2021, 16:29 (1374 days ago) @ Joe Duplessie (SNIPE 316)

- No text -

Back to the forum index
RSS Feed of thread