How do I split a combo list in a large text file?

Question

my problem is that I have a very large database of emails and passwords and I need to send it to a mysql database.

The .txt file format is something like this:

[email protected]:password1
[email protected]:password2
[email protected]:password3
[email protected]:password4
[email protected]:password5

My idea is to make a loop that takes the line and make it a variable, search the ":" and pick the text before, send it to the db and then the same with the after part of the line. How do I do this?

what is your problem? splitting? db-access? storage? file reading? — Patrick Artner, Nov 29 '18 at 23:02

Talip Tolga Sarı · Answer 1 · 2018-11-30T09:49:36.087

1

This can be done through simple split() method of the strings in python.

>>> a = '[email protected]:password1'
>>> b = a.split(':')
>>> b
['[email protected]', 'password1']

To accomodate @PatrickArtner's complex password fail this can be done:

atLocation = a.find('@')
realSeperator = atLocation + a[atLocation:].find(':')
emailName = a[0:atLocation]
emailDomain = a[atLocation:realSeperator]
email = emailName + emailDomain
password = a[realSeperator + 1:]

print(email, password)

>>> [email protected] com:plex:PassWord:fail

str.find() returns the first occurrence location of the given character in the given string. Emails can have : in their name field but they can not have @. So first locating the @ then locating the : would give you the correct separation locations. After that splitting the string will be piece of cake.

edited Nov 30 '18 at 09:49

answered Nov 29 '18 at 23:01

Talip Tolga Sarı

130
8

1

`'[email protected]:com:plex:PassWord:fail'` – Patrick Artner Nov 29 '18 at 23:16
1

`b = a.split(':',1)` ? https://docs.python.org/3/library/stdtypes.html#str.split - I was more concerned about a password having a `:` in it .. I am unsure wether emails can actually contain a `:` - seems they could if quoted: https://stackoverflow.com/questions/2049502/what-characters-are-allowed-in-an-email-address – Patrick Artner Nov 30 '18 at 09:55
1

from [wiki page](https://en.wikipedia.org/wiki/Email_address) it says this about the local part so I made sure first ':' is not in the local part "space and "(),:;<>@[\] characters are allowed with restrictions (they are only allowed inside a quoted string, as described in the paragraph below, and in addition, a backslash or double-quote must be preceded by a backslash);" However your solution is more elegant then mine – Talip Tolga Sarı Nov 30 '18 at 10:01
1

but the simple split(":",1) is less exact - it does not allow for any ":" in the local part even if quoted - yours got my upvote – Patrick Artner Nov 30 '18 at 10:03

Patrick Artner · Accepted Answer · 2018-11-30T10:08:27.383

Short program with some error handling:

Create demo data file:

t = """
[email protected]:password1
[email protected]:password2
[email protected]:password3
[email protected]:password4
[email protected]:password5
k
: """

with open("f.txt","w") as f: f.write(t)

Parse data / store:

def store_in_db(email,pw):
    # replace with db access code 
    # see    http://bobby-tables.com/python
    # for parametrized db code in python (or the API of your choice)
    print("stored: ", email, pw)


with open("f.txt") as r:
    for line in r:
        if line.strip():  # weed out empty lines
            try:
                email, pw = line.split(":",1) # even if : in pw: only split at 1st :
                if email.strip() and pw.strip(): # only if both filled
                    store_in_db(email,pw)
                else:
                    raise ValueError("Something is empty: '"+line+"'")

            except Exception as ex:
                print("Error: ", line, ex)

Output:

stored:  [email protected] password1

stored:  [email protected] password2

stored:  [email protected] password3

stored:  [email protected] password4

stored:  [email protected] password5

Error:  k
 not enough values to unpack (expected 2, got 1)
Error:  :  Something is empty: ': '

Edit: According to What characters are allowed in an email address? - a ':' may be part of the first part of an email if quoted.

This would theoretically allow inputs as

`"Cool:[email protected]:coolish_password"`

which will get errors with this code. See Talip Tolga Sans answer for how to break down the splitting differently to avoid this problem.

score -1 · Answer 3 · answered Nov 29 '18 at 23:08

Open file as context manager (with open(...)), You can iterate over the lines with a for loop, then regex match(re Module)(or just split on ":") and use sqlite3 to insert your values to DB.

So the file:

with open("file.txt", "r") as f:
    for line in f:
        pass #manipulation

Sqlite3 Docs: https://docs.python.org/2/library/sqlite3.html

How do I split a combo list in a large text file?

3 Answers3