maemo.org - Talk - View Single Post - Is there anyway to download a whole thread

	marxian	2011-10-11 , 17:19
	Posts: 2,448 \| Thanked: 9,523 times \| Joined on Aug 2010 @ Wigan, UK	#8

Originally Posted by ziggadebo

Ok,

Can someone point me in the right direction, I'm trying to run the python code, the below is where I'm at. I think I've defined the variables correctly?

I get an error 'test' is not defined, how do I correctly assign file name variable?

I'm not after the full solution just a friendly pointer in the right direction.

Thanks

Your code contains the following errors:

1. The fileName variable should be a string, i.e. "test". Your code attempts to assign the value of an undefined variable test to the variable fileName.

2. The link variable should also be a string. Your code attempts to perform urllib.urlopen() on an object that has been returned by that method (as part of a string).

3. You have added ".html" to the thread link. The link should end after the t parameter, i.e "http://talk.maemo.org/showthread.php?t=73315".

Solution:

Code:

#!/usr/bin/python

import io
import urllib
import sys

file_name = "test"
link = "http://talk.maemo.org/showthread.php?t=73315"
start_page = 1
end_page = 255

def get_thread(file_name, link, start_page, end_page):
    for num in range(int(start_page), int(end_page) + 1):
        #print "grabbing page " + str(num)
        page = urllib.urlopen("%s&page=%d" % (link, num)).read()
        with io.open("%s-page%d.html" % (file_name, num) , 'w') as file:
            file.write(unicode(page, 'utf-8'))
            file.close()

if __name__ == '__main__':
    sys.exit(get_thread(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]))

In my earlier example, I had a mixture of styles for the variable names, so I changed fileName to file_name.

__________________
'Men of high position are allowed, by a special act of grace, to accomodate their reasoning to the answer they need. Logic is only required in those of lesser rank.' - J K Galbraith

My website

GitHub

Last edited by marxian; 2011-10-11 at 17:34.

Quote & Reply |

The Following User Says Thank You to marxian For This Useful Post:
ziggadebo