[PYTHON] MultiDict: When doing form POST with a crawler, etc.

MultiDict

There can be multiple choices, so try using MultiDict from werkzeug:

$ pip install werkzeug

From the list dict:

>>> from werkzeug.datastructures import MultiDict
>>> MultiDict([('a',1), ('a', 2), ('a', 3)])
MultiDict([('a', 1), ('a', 2), ('a', 3)])

add to:

>>> d.update({'a': '4'})
>>> d
MultiDict([('a', 1), ('a', 2), ('a', 3), ('a', '4')])

update

>>> d['a']=5
>>> d
MultiDict([('a', 5)])

Get form data with BeautifulSoup

Get form from HTML with BeautifulSoup select ():

    def form_data(self, form_node):
        ''' form_node :Select from HTML()Form node
        '''

        data = MultiDict(list(
           (t['name'], t.get('value', '')) 
           for i in ['text', 'hidden', 'password', ]
           for t in form_node.select('input[type=%s]' % i)
        ) + list(
           (t['name'], t.text) 
           for t in form_node.select('textarea')
        ) + list(
           (t['name'], t.get('value', '')) 
           for t in form_node.select('input[checked]' % i)
        ) + list(
            (s.parent['name'], s.get('value', s.text))
            for s in form_node.select('option[selected]')
        ))  

        # TODO: "file", "image", 
        # TODO: HTML5 -- input(list) and datalist

        return data

When POSTing with requests, only the first parameter is passed

requests says that it supports MultiDict, but it doesn't work in my environment. .. ..

It can't be helped, so I converted it to dict for the time being:

>>> dict(d.lists())
{'a': [1, 2, 3, '4']}

    def post(self, uri, data):

        if isinstance(data, MultiDict):
            data = dict(data.lists())

        self.res = self.req.post(
            uri, data=data,
            verify=False, allow_redirects=False)
        self.last_uri = uri