Pandas: append dataframe to another df

I have a problem with appending of dataframe. I try to execute this code

df_all = pd.read_csv('data.csv', error_bad_lines=False, chunksize=1000000)
urls = pd.read_excel('url_june.xlsx')
substr = urls.url.values.tolist()
df_res = pd.DataFrame()
for df in df_all: for i in substr: res = df[df['url'].str.contains(i)] df_res.append(res)

And when I try to save df_res I get empty dataframe.df_all looks like

ID,"url","used_at","active_seconds"
b20f9412f914ad83b6611d69dbe3b2b4,"",2015-10-01 00:00:25,1
b20f9412f914ad83b6611d69dbe3b2b4,"",2015-10-01 00:00:31,30
f85ce4b2f8787d48edc8612b2ccaca83,"",2015-10-01 00:01:49,2
d3b0ef7d85dbb4dbb75e8a5950bad225,"",2015-10-01 00:03:19,34
078d388438ebf1d4142808f58fb66c87,"",2015-10-01 00:03:48,2
d3b0ef7d85dbb4dbb75e8a5950bad225,"",2015-10-01 00:04:21,4
d3b0ef7d85dbb4dbb75e8a5950bad225,"",2015-10-01 00:04:25,1
d3b0ef7d85dbb4dbb75e8a5950bad225,"",2015-10-01 00:04:26,9

and urls looks like

url

When I print res in a loop it doesn't empty. But when I try print in a loop df_res after append, it return empty dataframe. I can't find my error. How can I fix it?

2 Answers

If you look at the documentation for pd.DataFrame.append

Append rows of other to the end of this frame, returning a new object. Columns not in this frame are added as new columns.

(emphasis mine).

Try

df_res = df_res.append(res)

Incidentally, note that pandas isn't that efficient for creating a DataFrame by successive concatenations. You might try this, instead:

all_res = []
for df in df_all: for i in substr: res = df[df['url'].str.contains(i)] all_res.append(res)
df_res = pd.concat(all_res)

This first creates a list of all the parts, then creates a DataFrame from all of them once at the end.

If we want append based on index:

df_res = pd.DataFrame(data = None, columns= df.columns)
all_res = []
d1 = df.ix[index-10:index-1,] #it will take 10 rows before i-th index
all_res.append(d1)
df_res = pd.concat(all_res)