Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
T
Twint
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Locked Files
Issues
0
Issues
0
List
Boards
Labels
Service Desk
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Security & Compliance
Security & Compliance
Dependency List
License Compliance
Packages
Packages
List
Container Registry
Analytics
Analytics
CI / CD
Code Review
Insights
Issues
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
nanahira
Twint
Commits
5a8b681d
Commit
5a8b681d
authored
Oct 28, 2018
by
Francesco Poldi
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Revert "Added option to save user-info into elasticsearch"
This reverts commit 0606e981ecba7eac057e0b4496e4728761b8bb65.
parent
e6767bdb
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
11 deletions
+8
-11
Twint.py
Twint.py
+1
-1
twint/output.py
twint/output.py
+7
-10
No files found.
Twint.py
View file @
5a8b681d
...
@@ -162,7 +162,7 @@ def options():
...
@@ -162,7 +162,7 @@ def options():
ap
.
add_argument
(
"--user-full"
,
ap
.
add_argument
(
"--user-full"
,
help
=
"Collect all user information (Use with followers or following only)."
,
help
=
"Collect all user information (Use with followers or following only)."
,
action
=
"store_true"
)
action
=
"store_true"
)
ap
.
add_argument
(
"--user-info"
,
help
=
"Scrape user's info in tweet"
,
action
=
"store_
tru
e"
)
ap
.
add_argument
(
"--user-info"
,
help
=
"Scrape user's info in tweet"
,
action
=
"store_
fals
e"
)
ap
.
add_argument
(
"--profile-full"
,
ap
.
add_argument
(
"--profile-full"
,
help
=
"Slow, but effective method of collecting a user's Tweets and RT."
,
help
=
"Slow, but effective method of collecting a user's Tweets and RT."
,
action
=
"store_true"
)
action
=
"store_true"
)
...
...
twint/output.py
View file @
5a8b681d
...
@@ -82,20 +82,17 @@ async def tweetUserData(tweet,config, conn):
...
@@ -82,20 +82,17 @@ async def tweetUserData(tweet,config, conn):
user_ids
=
set
()
user_ids
=
set
()
usernames
=
[]
usernames
=
[]
for
user
in
tweet
.
mentions
:
for
user
in
tweet
.
mentions
:
if
config
.
Database
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
user_ids
.
add
(
user
[
"id"
])
user_ids
.
add
(
user
[
"id"
])
usernames
.
append
(
user
[
"screen_name"
])
usernames
.
append
(
user
[
"screen_name"
])
for
user
in
tweet
.
tags
:
for
user
in
tweet
.
tags
:
if
config
.
Database
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
user_ids
.
add
(
user
[
"id"
])
user_ids
.
add
(
user
[
"id"
])
usernames
.
append
(
user
[
"screen_name"
])
usernames
.
append
(
user
[
"screen_name"
])
for
user
in
tweet
.
replies
:
for
user
in
tweet
.
replies
:
if
config
.
Database
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
if
db
.
get_user_id
(
conn
,
user
[
"id"
])
==
-
1
and
user
[
"id"
]
not
in
user_ids
:
user_ids
.
add
(
user
[
"id"
])
user_ids
.
add
(
user
[
"id"
])
usernames
.
append
(
user
[
"screen_name"
])
usernames
.
append
(
user
[
"screen_name"
])
for
user
in
usernames
:
for
user
in
usernames
:
url
=
f
"http://twitter.com/{user}?lang=en"
url
=
f
"http://twitter.com/{user}?lang=en"
await
get
.
User
(
url
,
config
,
conn
)
await
get
.
User
(
url
,
config
,
conn
)
...
@@ -106,7 +103,7 @@ async def Tweets(tw, location, config, conn):
...
@@ -106,7 +103,7 @@ async def Tweets(tw, location, config, conn):
if
copyright
is
None
and
is_tweet
(
tw
):
if
copyright
is
None
and
is_tweet
(
tw
):
tweet
=
Tweet
(
tw
,
location
,
config
)
tweet
=
Tweet
(
tw
,
location
,
config
)
if
(
config
.
Database
or
config
.
Elasticsearch
)
and
config
.
User_info
:
if
config
.
Database
is
not
None
and
config
.
User_info
:
await
tweetUserData
(
tweet
,
config
,
conn
)
await
tweetUserData
(
tweet
,
config
,
conn
)
if
datecheck
(
tweet
.
datestamp
,
config
):
if
datecheck
(
tweet
.
datestamp
,
config
):
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment