|
Post by microfarad on May 24, 2010 2:56:50 GMT
Well, it is fairly clear at this point that the ODBFs will not be deleted. But, maybe we want to move here anyways... It may be time.
Here is my backup plan: We need a list of board names. In the URL you have a parameter called "board". Example: "board=about"
The second parameter is "action". To view raw post data use "action=print". This will give you all of the posts for all of the pages.
The last parameter is "thread". Example: "thread=2"
We need to write a program which will somehow find the valid board name / thread number pairs and store the post data to a file structure. It would be impractical to do it by trial and error since we don't even know the highest thread number (it is higher than the number of threads).
Help?
|
|
|
Post by Phantom Zero on May 24, 2010 3:44:05 GMT
i like this idea. backup just incase molopoy deleats the forum or listo deleats the forum.
|
|
|
Post by microfarad on May 24, 2010 3:51:56 GMT
Molopoy? Ha! He's gone. It's Listo I'm not so sure about...
|
|
|
Post by Qwerty on May 24, 2010 4:40:38 GMT
And Lag, don't forget Lag.
Of course, threads are vanishing over there, which makes backups even more imperative.
|
|
|
Post by Vertigo on May 24, 2010 21:19:06 GMT
Right now the highest thread number is 7815, but it'll change before we get this thing going.
|
|
|
Post by Qwerty on May 24, 2010 22:46:42 GMT
Yup. Probably. I mean, it changes with each thread.
|
|
|
Post by Disabled on May 26, 2010 19:36:53 GMT
So without reading this thread I wrote a small programm that backs up a whole proboards forum. (I hope its universal.) Just enter the base url and it does the rest. I'm fixing a few bugs right now, but it should be good to go in no time. I could create static pages from the data and host them on my page (if I have enough space). Sad that we stayed at proboards, because with a real forum, we could just import all the data and keep all threads. But since the layout is more important then the content, we will have to work with this workaround. If you can disable the security check (not that useless JS you use, the real thing) for not logged in users, I could perhaps create a bot that recreates all the threads. Or I could register a bot to the forum who creates all the threads, but that would be very weired...
|
|
|
Post by nmagain on May 26, 2010 20:39:13 GMT
Isn't Listo and Lag the same person?
|
|
|
Post by Disabled on May 27, 2010 0:56:01 GMT
So the backup is complete now. I have to sanity check the result, but it should be ok now. I have it stored as a JSON "database". Every thread has its title, its forum and its posts children. Every post has the author, the date, (again) the title and the post itself. Here is a quick and dirty version of the forum. The Forum names are still like the internal names, perhaps I will change that later. But no guarantees. The threads are probably not sorted, I don't know. Some threads are missing, if they became larger then 500kb (the limit of my host). I have them here, but I can't upload them. If needed, I can upload specific missing threads. The usernames are your login names. I don't care if you use a really stupid login name, so I won't change that. Inter-Forum Links won't work. Whatever I forgot...
|
|
|
Post by Disabled on May 27, 2010 18:24:18 GMT
So the old forum got deleted? Thats sad, I just tried to start a new crawler run, that also copies the Forum names and a little bit more... Looks like this is all we get. I hope I made no mistakes. I still have the forum as a db, so I can change stuff in the static forum I created. For instance I could count the posts for members if you want...
|
|
|
Post by Qwerty on May 27, 2010 23:25:20 GMT
|
|
|
Post by disabled on May 27, 2010 23:39:26 GMT
I wonder if I can still enter that 100 lines of code competition... the crawler is just about 100 lines, and the program generating the static forum is around 50 lines
|
|
|
Post by Qwerty on May 28, 2010 0:12:45 GMT
Heh, you probably still can. Although it was deleted. You can re-make the thread, I guess. It would mean a second entry.
|
|
|
Post by ~Memzak~ on May 28, 2010 11:56:08 GMT
OMIGAWSH.....
I LOVE YOU DISABLED I LOVE YOU SO VERY MUCH I can't believe this static copy! I LOVE IT.
I'm going through the whole thingeh. See what I can bring back to this forum. So that it is never Forgotten
EDIT: Sorry staff, it looks like it didn't copy any of the staff, high staff, contest and hidden staff boards. but it's still good anyways.
|
|
|
Post by disabled on May 28, 2010 12:19:37 GMT
Nope, If the forum would have stayed a little longer, I would have asked to make them public or I might have (if I was very bored) implement a basic authentication support. But since the forum is down now, all the hidden and locked boards are gone. And yeah, I was looking for the contest board too and didn't find it. I thought it might have a weired name or similar. Do you know the internal name of the contest board? If you have ideas what how I should restructure the forum, just tell me. If you want to properly name the forums, you can help me: Make a list which maps the board name (what you see) to its real name. Ideally structure it like this:
{"help": "Welcome: Please Read.", "helpprovided": "Help", "site" : "About the Site", "announce" : "Announcements", "powder" : "Powder Game", "stickranger": "Stick Ranger", "other" : "Other", "gentalk" : "General Talk", "memboards" : "Member Boards", "publicarchives" : "Public Archives", "ideas" : "Contest Ideas", "con" : "Contests", "shop" : "Dan-Ball Shop", "requests" : "Official Graphic Artist Shop", "badges" : "Badges", "memgames" : "Game Internals", "memgen" : "Member's Lounge"} Because thats what I'm using for renaming the base forums. If you want some markup in the forums, I can do that as well. Just ask. I can also release the whole database with the program, if someone is interested. But the program is very messy to read because I was lazy.
|
|
|
Post by Qwerty on May 28, 2010 20:06:33 GMT
There was nothing important in the staff boards.
From here, it seems to be a matter of structure. It should be easy enough to find the board name from the board ID, and the subboards are fairly simple.
|
|
|
Post by ~Memzak~ on May 31, 2010 14:14:29 GMT
'ere we go! I sorted out all the board names.
{"help": "Welcome: Please Read.", "helpprovided": "Help", "site" : "About the Site", "announce" : "Announcements", "powder" : "Powder Game", "stickranger": "Stick Ranger", "other" : "Other", "gentalk" : "General Talk", "memboards" : "Member Boards", "publicarchives" : "Public Archives", "ideas" : "Contest Ideas", "con" : "Contests", "shop" : "Dan-Ball Shop", "requests" : "Official Graphic Artist Shop", "badges" : "Badges", "memgames" : "Game Internals", "memgen" : "Member's Lounge"}
Changes to:
{"shop: "Dan-Ball Shop", "help": "Welcome: Please Read", "site": "About the Site", "publicarchives": "Public Archives", "powder": "Powder Game", "requests": "Official Graphic Artist Shop", "gentalk": "General Talk", "sticktranger": "Stick Ranger", "other": "Other", "badges": "Badges", "debate": "Debate Board", "ham": "Ham", "spam": "Spam Board", "rocats": "Rocats (Mobile)", "gam": "Forum Games", "announcevital": "Important Anouncments", "respect": "Respect Badge Requests", "pghelp": "Powder Game Help", "pgtech": "Powder Game Technology", "srhelp": "Stick Ranger Help", "stick": "Irritation Stickman", "catshot": "Catshot (Mobile)", "upload": "Uploads", "stories": "Stories", "planetsimmobile": "Planet Simulation (Mobile)", "raytracer": "Ray Trace Fighter", "art": "Art Board", "announcefun": "Fun Announcments", "helpprovided": "Help", "panda": "Panda", "announcesite": "Forum Updates", "sandmoji": "Sand Moji", "minetower": "Minetower", "genchat": "General Chat", "announcesecurity": "Security Information", "liquid": "Liquid Webtoy", "valistroke": "Valistoke (Mobile)", "tech": "Technology", "memgames" : "Game Internals", "memgen" : "Member's Lounge", "game": "Earth Editor", "idea": "Stick Ranger Ideas"}
Anything else needed?
I'd also be happy to sort out the boards out... and group the sub-boards with main boards and what-not.
|
|
|
Post by disabled on May 31, 2010 23:28:57 GMT
Thats a good start. Still missing are "pgideas", "stickranger" and "update". If you want them ordered, make a list like the following:
[["board", "subboard", "subboard"], ["board2", "subboard", "subboard"],... ]
|
|
|
Post by ~Memzak~ on Jun 1, 2010 7:24:32 GMT
Oh ok, but for the boards and suboards must I names the boards by their URL name or their changed name? Or both?
Anyways, here's the updated list of Names (with pgideas, stickranger and update)
{"shop: "Dan-Ball Shop", "help": "Welcome: Please Read", "site": "About the Site", "publicarchives": "Public Archives", "powder": "Powder Game", "requests": "Official Graphic Artist Shop", "gentalk": "General Talk", "stickranger": "Stick Ranger", "other": "Other Games", "badges": "Badges", "debate": "Debate Board", "ham": "Ham Race", "spam": "Spam Board", "rocats": "Rocats (Mobile)", "gam": "Forum Games", "announcevital": "Important Anouncments", "respect": "Respect Badge Requests", "pghelp": "Powder Game Help", "pgtech": "Powder Game Technology", "srhelp": "Stick Ranger Help", "stick": "Irritation Stickman", "catshot": "Catshot (Mobile)", "upload": "Uploads", "stories": "Stories", "planetsimmobile": "Planet Simulation (Mobile)", "raytracer": "Ray Trace Fighter", "art": "Art Board", "announcefun": "Fun Announcments", "helpprovided": "Help", "panda": "Micro Panda", "announcesite": "Forum Updates", "sandmoji": "Sand Moji", "minetower": "Minetower", "genchat": "General Chat", "announcesecurity": "Security Information", "liquid": "Liquid Webtoy", "valistroke": "Valistoke (Mobile)", "tech": "Technology", "memgames" : "Game Internals", "memgen" : "Member's Lounge", "game": "Earth Editor", "idea": "Stick Ranger Ideas", "pgideas": "Accepted Powder Game Elements", "update": "Stick Ranger Updates"}
If it is just for the board URL Names then here is the list:
[["help"], ["helpprovided"], ["site"], ["announcevital", "announcesite","announcesecurity", "announcefun"], ["powder", "pgideas", "pghelp", "pgtech"], ["sticktranger", "update", "srhelp", "idea"], ["other", "game", "stick", ", "raytracer", "liquid", "panda", "sandmoji", "ham", "minetower", "catshot", "rocats", "planetsimmobile", "valistroke"], ["gentalk", "genchat", "upload", "tech", "debate", "stories", "art"], ["gam"], ["spam"], ["publicarchives"], ["shop"], ["badges", "respect"], ["requests"], ["memgames"], ["memgen"], ]
If that is it then what about catagories? Maybe Or maybe just make an empty line between categories just to separate them.
|
|
|
Post by Qwerty on Jun 1, 2010 7:42:50 GMT
Quick question: Disabled, did you manage to get the member boards?
|
|
|
Post by disabled on Jun 1, 2010 9:20:36 GMT
Memzak, that was about what I wanted. I also added the categories. I hope thats what you wanted. If not, just post your comments. Also if you want to make it more advanced, with better styles and such: If you know CSS, you can create a stylesheet and I will upload it. I can also give every link a class tag so you can reference them easily.
Qwerty, it seems I didn't. Weired, I'd really like to know why they didn't get downloaded... after all, my complete backup isnt so complete.
|
|
|
Post by ~Memzak~ on Jun 1, 2010 10:41:26 GMT
Cool, thanks for the Catagories it really helped. Just some of the sub-boards were not put in... (weird I'll look in to it)
Would it be okay, if I just edit the plain HTML source code for it instead, it is much more familiar to me.... if possible....
|
|
|
Post by disabled on Jun 1, 2010 13:21:08 GMT
For things like background and stuff like that, just edit the html. If you want to relayout the links and everything it would probably be easier if you just tell me what to do, because the file gets generated automatically. So I don't have to edit each line on its own, just once. You can do the same to the Forum indexes and the thread files if you want.
|
|
|
Post by ~Memzak~ on Jun 1, 2010 15:39:38 GMT
Oh ok, cool... sorta....
because I've been working on reworking the whole HTML (and making pwetty graphics for it)
Anyways.... maybe it would be even easier if I had direct access to the HTML... Anyways, here is the current update. (Added nice graphics and stuff) [Bear in mind my programming is still very rusty, I haven't used it in a while]
<html><head>
<style type="text/css"> #image { background: url(http://i47.tinypic.com/166ws5s.png) repeat-y; } #image1 { background: url(http://i48.tinypic.com/2anac1.png) repeat-x; } </style>
<link rel="icon" href="http://i45.tinypic.com/f3wcuf.png">
</head><body bgcolor="#5f2c12"> <div id="image1"> <img src="http://i46.tinypic.com/2vdo1fk.png" /> <div id="image">
<br><br><img src="http://i49.tinypic.com/2a8g9is.png" /><br> <a href="help">Welcome: Please Read</a><br> <a href="helpprovided">Help</a><br> <a href="site">About the Site</a><br> <a href="announcevital">Important Anouncments</a><br> <a href="announcesite">Forum Updates</a><br> <a href="announcesecurity">Security Information</a><br> <a href="announcefun">Fun Announcments</a><br> <br><img src="http://i50.tinypic.com/qrx17p.png" /><br> <a href="powder">Powder Game</a><br> <a href="pgideas">Powder Game Accepted Elements</a><br> <a href="pghelp">Powder Game Help</a><br> <a href="pgtech">Powder Game Technology</a><br> <a href="stickranger">Stick Ranger</a><br> <a href="update">Stick Ranger Updates</a><br> <a href="srhelp">Stick Ranger Help</a><br> <a href="idea">Stick Ranger Ideas</a><br> <a href="other">Other Games</a><br> <a href="game">Earth Editor</a><br> <a href="stick">Irritation Stickman</a><br> <a href="raytracer">Ray Trace Fighter</a><br> <a href="liquid">Liquid Webtoy</a><br> <a href="panda">Micro Panda</a><br> <a href="sandmoji">Sand Moji</a><br> <a href="ham">Ham Race</a><br> <a href="minetower">Minetower</a><br> <a href="catshot">Catshot (Mobile)</a><br> <a href="rocats">Rocats (Mobile)</a><br> <a href="planetsimmobile">Planet Simulation (Mobile)</a><br> <a href="valistroke">Valistoke (Mobile)</a><br> <br><img src="http://i45.tinypic.com/9itwkx.png" /><br>  <a href="gentalk">General Talk</a><br> <a href="genchat">General Chat</a><br> <a href="upload">Uploads</a><br> <a href="tech">Technology</a><br> <a href="debate">Debate Board</a><br> <a href="stories">Stories</a><br> <a href="art">Art Board</a><br> <a href="gam">Forum Games</a><br> <a href="spam">Spam Board</a><br> <a href="publicarchives">Public Archives</a><br> <br><img src="http://i48.tinypic.com/bebqfa.png" /><br> <a href="shop">Dan-Ball Shop</a><br> <a href="badges">Badges</a><br> <a href="respect">Respect Badge Requests</a><br> <a href="requests">Official Graphic Artist Shop</a><br> <br><img src="http://i50.tinypic.com/k3u54h.png" /><br> <a href="memgames">Game Internals</a><br> <a href="memgen">Member's Lounge</a><br> <br> <br> </div></div>
How do you make a certain amount of spaces fit into one space? Meh... until then I did that repeat thing as you can see above (so that the graphics don't look f-ed up)
Anyways, I hope that it is (mostly) bug free...
|
|
|
Post by disabled on Jun 1, 2010 23:37:05 GMT
Because I think the forum header waaaay to big, I changed it a big. (Finally I can do it, at least for the backup ) But I somehow don't understand what you mean with "direct access to the html" and "fit certain amount of spaces into one space".
|
|
|
Post by ~Memzak~ on Jun 2, 2010 10:06:54 GMT
Nah, I was just getting a bot annoyed at typing "&"n"b"s"p";" so many times... (without inverted comma's)
Anyways, I love the new header, it's much more fitting for the backup. What did you use to edit it?
|
|
|
Post by disabled on Jun 2, 2010 10:17:48 GMT
I used Gimp to edit it. I don't like gimp very much, but I'm with Linux, so there are few alternatives. I would have been much faster if I had used Paint.net under windows. And for the nbsp thing, there is this cool invention called copy&paste...
|
|
|
Post by ~Memzak~ on Jun 2, 2010 10:47:12 GMT
Lol, I know.... and I used copy+paste, but it still took a while...
Anyways... GIMP.... hmm, I thought so, that's what I use... I like it though, maybe it is just personal preference.
Anyways, should we add anything more, to make the site nicer? Or just leave it the way it is? (I've been thinking of maybe adding profiles by using a program to count all posts made by (enter-user-name-here) and allow for quicker access to find out how many posts you had. Although I'd have no idea where to start for that....
What do you think?
|
|
|
Post by Qwerty on Jun 2, 2010 13:27:37 GMT
Well, you have the main title okay, but what about the boards and threads? Those are still white, and backed up, but they don't fit.
|
|
|
Post by ~Memzak~ on Jun 3, 2010 7:58:56 GMT
Prehaps we should make a global code that makes all the thread like that. Unfortunately I have no idea where to start in doing this.
Besides I'm not sure how to make the writing inside of the graphics without adding the code for a space like 50 times so that the writing is in the graphic background.
Any help here?
|
|
|