Skip to content

T
Twint
Commit 88a22f51 authored by Francesco Poldi's avatar Francesco Poldi Committed by GitHub
Browse files
Options

Merge pull request #155 from twintproject/master 

Merging master into dev
parents b74d7e46 a72347f4
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 18 additions and 3532 deletions
README.md
View file @ 88a22f51
...@@ -21,9 +21,9 @@ Some of the benefits of using Twint vs Twitter API: ...@@ -21,9 +21,9 @@ Some of the benefits of using Twint vs Twitter API:
- `pip3 install -r requirements.txt` - `pip3 install -r requirements.txt`
## Installing ## Installing
- **Git**: `git clone https://github.com/haccer/twint.git` - **Git**: `git clone https://github.com/twintproject/twint.git`
- **Pip**: `pip3 install twint` - **Pip**: `pip3 install twint`
- **With MySQL**: `git clone -b mysql --single-branch https://github.com/haccer/twint.git twint-mysql` - **With MySQL**: `git clone -b mysql --single-branch https://github.com/twintproject/twint.git twint-mysql`
## Basic Examples and Combos. ## Basic Examples and Combos.
A few simple examples to help you understand the basics: A few simple examples to help you understand the basics:
...@@ -48,10 +48,10 @@ A few simple examples to help you understand the basics: ...@@ -48,10 +48,10 @@ A few simple examples to help you understand the basics:
- `python3 twint.py -u username --profile-full` - Use a slow, but effective method to gather all the Tweets from a user's profile (Including Retweets). - `python3 twint.py -u username --profile-full` - Use a slow, but effective method to gather all the Tweets from a user's profile (Including Retweets).
- `python3 twint.py -u username --retweets` - Use a quick method to gather the last 900 Tweets (that includes retweets) from a user's profile. - `python3 twint.py -u username --retweets` - Use a quick method to gather the last 900 Tweets (that includes retweets) from a user's profile.
More detail about the commands and options are located in the [wiki](https://github.com/haccer/twint/wiki/Commands) More detail about the commands and options are located in the [wiki](https://github.com/twintproject/twint/wiki/Commands)
## Using Twint as a Module (Recommended) ## Using Twint as a Module (Recommended)
Twint can now be used as a module and supports custom formatting. **More details are located in the [wiki](https://github.com/haccer/twint/wiki/Module)** Twint can now be used as a module and supports custom formatting. **More details are located in the [wiki](https://github.com/twintproject/twint/wiki/Module)**
#### Example #### Example
```python ```python
...@@ -79,19 +79,15 @@ twint.run.Search(c) ...@@ -79,19 +79,15 @@ twint.run.Search(c)
- MySQL (See MySQL Branch) - MySQL (See MySQL Branch)
### Elasticsearch Setup ### Elasticsearch Setup
Details on setting up Elasticsearch with Twint is located in the [wiki](https://github.com/haccer/twint/wiki/Elasticsearch). Details on setting up Elasticsearch with Twint is located in the [wiki](https://github.com/twintproject/twint/wiki/Elasticsearch).
### Graph Visualization ### Graph Visualization
![graph](https://i.imgur.com/EEJqB8n.png) ![graph](https://i.imgur.com/EEJqB8n.png)
[Graph](https://github.com/haccer/twint/tree/master/graph) details are also located in the [wiki](https://github.com/haccer/twint/wiki/Graph). [Graph](https://github.com/twintproject/twint/tree/master/graph) details are also located in the [wiki](https://github.com/twintproject/twint/wiki/Graph).
We are testing a (free) graph plugin for Kibana, details located in the Wiki! We are testing a (free) graph plugin for Kibana, details located in the Wiki!
## Thanks
Thanks to [@hpiedcoq](https://github.com/hpiedcoq) & [@pielco11](https://github.com/pielco11) for contributing several features!
## Contact ## Contact
Shout me out on Twitter: [@now](https://twitter.com/now)
If you have problems or have suggestions don't hesitate to open an issue or ask about it directly. If you have any questions, want to join in on discussions, or need extra help, you are welcome to join our OSINT focused [Slack server](https://os-int.slack.com/join/shared_invite/enQtMzc4NzY5ODI3NDI3LWRlOGNhN2U3OTUwY2Q1OTk5MDI2YjliOWQ1OTI5NzAyZjc0MDhiYTQ3NTY4MjMxY2E0MTRhOTVlN2M0ZmJhMjI).
graph/Dockerfile deleted 100644 → 0
View file @ b74d7e46
# Twint OSINT Explorer - https://github.com/haccer/twint/tree/master/graph
# $ xhost local:root
# docker run --name twint \
# -v /tmp/.X11-unix:/tmp/.X11-unix \
# -e DISPLAY=unix$DISPLAY \
# --rm <image>
FROM node:9.11.1-stretch
LABEL maintainer "Cody Zacharias <codyzacharias@pm.me>"
# Install Packages
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y \
libsqlite3-dev \
libxss1 \
libx11-xcb-dev \
libxtst-dev \
libgconf-2-4 \
libnss3 \
libasound-dev
WORKDIR /data
# Install node-sqlite3
RUN git clone https://github.com/mapbox/node-sqlite3.git && \
cd node-sqlite3 && \
npm install --build-from-source
WORKDIR /data
# Install Twint
RUN git clone https://github.com/haccer/twint.git && \
cd twint/graph && \
npm install
# Make sure we're in the right directory
WORKDIR /data/twint/graph
CMD ["npm", "start"]
graph/README.md deleted 100644 → 0
View file @ b74d7e46
# Graph Visualization How-To
![graph](https://i.imgur.com/EEJqB8n.png)
Under `Graph` directory there is the source code of the `Twint OSINT Explorer`, the compiled version will be provided.
## Install
#### Dependencies
- NodeJS
- `libsqlite3-dev libxss1 libx11-xcb-dev libxtst-dev libgconf-2-4 libnss3 libasound-dev`
### Debian/Ubuntu Based Systems
```
chmod +x install.sh
./install.sh
```
### Docker
```
xhost local:root
docker run --name twint -v /tmp/.X11-unix:/tmp/.X11-unix -v $(PWD)/data:/data/data -e DISPLAY=unix$DISPLAY --rm c0dy/twint-explorer
```
### Other
Steps:
1. `Install node-sqlite3` - I recommend building this from source by doing the following:
```
git clone git clone https://github.com/mapbox/node-sqlite3.git
cd node-sqlite3
npm install --build-from-source
```
or you can run
```
npm install sqlite3
```
2. `npm install` - In this directory
3. To start `Twint OSINT Explorer` just run `npm start .`
## Descrption
On the left side there are:
### Home
Does nothing (now as now).
### Dashboard
You will have to create a file `dashboard.txt` in that directory, that file will contain the url of the `iframe` object of the Kibana Dashboard... this does nothing more than using your browser to visualize the dashboard that you made in Kibana.
### Graph
You will **have to** have the database to visualize users in a pretty nice graph.
How to:
1. `Database file`: the name file of the database (e.g.: twint.db);
2. `Graph file`: **useless** (now as now);
3. Select the table: `Users`, `Followers` or `Following`;
4. `Condition` the value that you want to graph, in case of the Users table this will graph that specific user (you can use * to graph every user that you scraped, this might slow down), the same for Followers and Following tables... given a specific condition it will load users with that name (in case of users table), users that have the "condition-user" as follower (in case of followers table) and the same for following table;
5. `Load Settings`: this will prepare the connection between users, does not plot;
6. `Load Graph`: plots;
7. You can use `Raw Query` to execute raw queries (e.g.: `select column from table where.....`).
`Import Graph` and `Export Graph` are useless, I'm working on a way to achieve this.
**Attention here**: using * in condition might require a lot of time, I did the best to speed up, good luck.
## Run
If you don't build you can run `npm start .` and if you want to build you can run `npm run build` (you might change configs in `package.json` to build for your own system and arch).
## Dev
This feature and this Wiki is highly under development. The code and features might not be completed but everything works as expected and tested.
graph/css/uikit.min.css deleted 100644 → 0
View file @ b74d7e46
This source diff could not be displayed because it is too large. You can view the blob instead.
graph/index.html deleted 100644 → 0
View file @ b74d7e46
<!DOCTYPE html>
<html>
<head>
<title>Twint OSINT Explorer</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="css/uikit.min.css" />
<script src="js/uikit.min.js"></script>
<script src="js/uikit-icons.min.js"></script>
<link rel="stylesheet" href="photon/css/photon.min.css">
<style>
body {
color: #d3d3d3;
font: 12pt arial;
min-height: 100px;
min-width: 100px;
}
#pane {
width: auto;
height: auto;
min-width: 400px;
min-height: 400px;
overflow:hidden;
}
#includedContent {
width: 100%;
height: 100%;
overflow: hidden;
}
</style>
</head>
<body>
<div class="window">
<div class="window-content">
<div class="pane-group">
<nav class="nav-group">
<h5 class="nav-group-title">Favorites</h5>
<a class="nav-group-item" id="loadHome">
<span class="icon icon-home"></span>
</a>
<span class="nav-group-item" id="dashboard">
<ul class="uk-iconnav uk-iconnav-vertical">
<li><a href="#" uk-icon="icon: image"> Dashboard </a></li>
</ul>
</span>
<span class="nav-group-item" id="twint">
<ul class="uk-iconnav uk-iconnav-vertical">
<li><a href="#" uk-icon="icon: social"> Graph </a></li>
</ul>
</span>
</nav>
<div class="pane" id="pane">
<div id="includedContent"></div>
<script>window.$ = window.jQuery = require('jquery');</script>
<script>
const {ipcRenderer} = require('electron');
$("#twint").click(function(){
$("#includedContent").load("pages/form.html");
});
$("#dashboard").click(function(){
$("#includedContent").load("pages/dashboard.html");
});
$(document).ready(function(){
$("#includedContent").load("pages/home.html");
});
$("#loadHome").click(function(){
$("#includedContent").load("pages/home.html");
});
</script>
</div>
</div>
</div>
</div>
</body>
</html>
\ No newline at end of file
graph/index.js deleted 100644 → 0
View file @ b74d7e46
//const electron = require('electron');
const url = require('url');
const path = require('path');
var fs = require('fs');
//var sqlite3 = require('sqlite3').verbose();
const {app, BrowserWindow, ipcMain, Menu, MenuItem} = require('electron');
const menu = new Menu()
menu.append(new MenuItem({ label: 'Hello' }))
menu.append(new MenuItem({ type: 'separator' }))
menu.append(new MenuItem({ label: 'Electron', type: 'checkbox', checked: true }))
let mainWindow;
app.on('ready' , function(){
win = new BrowserWindow({
width: 800,
height: 600,
minWidth: 200,
minHeight: 200,
})
win.on('closed', () => {
win = null
});
win.loadURL(url.format({
pathname: path.join(__dirname, 'index.html'),
protocol: 'file:',
slashes: true
}));
ipcMain.on('get-dashboard-url', (event, arg) => {
fs.readFile('./dashboard.txt', 'utf-8', (err, data) => {
event.sender.send('asynchronous-reply', data);
});
});
var graphName = "";
ipcMain.on('Name', (event, arg) => {
graphName = arg;
});
ipcMain.on('importData', (event, arg) => {
fs.readFile(graphName, 'utf-8', (err, data) => {
event.sender.send('responseGraph', data);
});
});
ipcMain.on('exportData', (event, arg) => {
fs.writeFile(graphName, arg)
});
ipcMain.on('show-context-menu', (event, arg) => {
const menu = new Menu();
menu.append(new MenuItem ({
label: 'Item ' + arg,
}));
menu.popup(menu)
});
});
graph/install.sh deleted 100644 → 0
View file @ b74d7e46
#!/bin/sh
# Installation script for Debian based systems
# Install Dependencies
sudo apt-get install -y libsqlite3-dev \
libxss1 \
libx11-xcb-dev \
libxtst-dev \
libgconf-2-4 \
libnss3 \
libasound-dev
# Install node-sqlite3
git clone git clone https://github.com/mapbox/node-sqlite3.git
cd node-sqlite3
npm install --build-from-source
# Build
cd ..
npm install
graph/js/uikit-icons.min.js deleted 100644 → 0
View file @ b74d7e46
This diff is collapsed.
graph/js/uikit.min.js deleted 100644 → 0
View file @ b74d7e46
This source diff could not be displayed because it is too large. You can view the blob instead.
graph/package-lock.json deleted 100644 → 0
View file @ b74d7e46
This diff is collapsed.
graph/package.json deleted 100644 → 0
View file @ b74d7e46
{
"name": "twint-explorer",
"version": "1.0.0",
"description": "Twint Explorer",
"main": "index.js",
"scripts": {
"start": "electron .",
"rebuild": "electron-rebuild -f -w sqlite3",
"build": "electron-packager . --all twint --ignore-modules=node_modules/electron-*"
},
"author": "Francesco Poldi @pielco11",
"license": "MIT",
"dependencies": {
"electron": "^1.8.4",
"jquery": "^3.3.1",
"uikit": "^3.0.0-beta.40",
"vis": "^4.21.0"
},
"devDependencies": {
"electron-osx-sign": "^0.4.10",
"electron-packager": "^12.0.1",
"electron-rebuild": "^1.7.3"
}
}
graph/pages/dashboard.html deleted 100644 → 0
View file @ b74d7e46
<html>
<link rel="stylesheet" href="photon/css/photon.min.css">
<body>
<script>
window.$ = window.jQuery = require('jquery');
//const {ipcRenderer} = require('electron');
$(document).ready(function(){
ipcRenderer.send('get-dashboard-url', 'asynchronous-message');
console.log("sent");
ipcRenderer.on('asynchronous-reply', (event, arg) => {
document.getElementById("dFrame").src = arg;
console.log(arg);
});
})
</script>
<iframe src="" height=100% width=100% id="dFrame"></iframe>
</body>
</html>
\ No newline at end of file
graph/pages/form.html deleted 100644 → 0
View file @ b74d7e46
<html>
<h1>Twint Graph Visualizer</h1>
<head>
<title>
Twint Graph Visualizer
</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="css/uikit.min.css" />
<script src="js/uikit.min.js"></script>
<script src="js/uikit-icons.min.js"></script>
<script>
const vis = require('vis');
</script>
<link href="node_modules/vis/dist/vis-network.min.css" rel="stylesheet" type="text/css"/>
<link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<link rel="stylesheet" href="http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css">
<style>
form {
margin-left: 10px;
}
h1 {
margin-left: 10px;
}
body {
color: black
}
#mynetwork {
width: 100%;
height: 100%;
overflow:hidden;
border: 1px solid #444444;
background-color: #444444;
}
</style>
</head>
<body>
<form action=ìì>
<div class="uk-margin" uk-margin>
<div uk-form-custom="target: true">
<input class="uk-input uk-form-width-medium" type="text" placeholder="Database file" id="submitDatabase">
</div>
<div uk-form-custom="target: true">
<input class="uk-input uk-form-width-medium" type="text" placeholder="Graph file" id="graphFile">
</div>
<div uk-form-custom="target: > * > span:first-child">
<select id="submitTable">
<option value="">Please select...</option>
<option value="users where user = ">Users</option>
<option value="followers where follower = ">Followers</option>
<option value="following where follows =">Following</option>
</select>
<button class="uk-button uk-button-default" type="button" tabindex="-1">
<span></span>
<span uk-icon="icon: chevron-down"></span>
</button>
</div>
<input class="uk-input uk-form-width-medium" type="text" id="submitQuery" placeholder="Condition">
</div>
<div>
<input class="uk-input uk-form-width-medium" type="text" id="submitRawQuery" placeholder="Raw Query">
<button type="button" class="uk-button uk-button-default" onclick="loadQuery()">Load settings</button>
<button type="button" class="uk-button uk-button-default" onclick="loadGraph()">Load graph</button>
<button type="button" class="uk-button uk-button-default" onclick="importNetwork()">Import Graph</input>
<button type="button" class="uk-button uk-button-default" onclick="exportNetwork()">Export Graph</input>
</div>
</form>
<script>
//var sqlite3 = require('sqlite3').verbose();
var network;
var data;
var nodes = new vis.DataSet();
var edges = new vis.DataSet();
var query = "select * from ";
var _table = "";
var _where = "";
var _raw_ = "";
var val = "";
const sqlite3 = require('sqlite3').verbose();
function loadQuery(){
_where = document.getElementById("submitQuery").value;
_table = document.getElementById("submitTable").value;
_raw_ = document.getElementById("submitRawQuery").value;
if (_raw_) { query = _raw_;}
else if (_where == "*") { query += _table.split(" ")[0];}
else{query += _table + "\"" + _where + "\"";}
val = document.getElementById("submitDatabase").value;
var db = new sqlite3.Database(val);
console.log(query);
if (query.split(" ")[3] == "followers") {
db.each(query, (err, result) => {
try{
nodes.add({
id : result.follower,
label: result.follower,
group: 'user'
});
}catch(err) {;}
try{
nodes.add({
id : result.user,
label: result.user,
group: 'user'
});
}catch(err) {;}
try{
edges.add({
id : result.user + "_" + result.follower,
from : result.follower,
to : result.user,
arrows: 'to'
});
}catch(err) {
edges.remove(result.user + "_" + result.follower);
edges.add({
id : result.user + "_" + result.follower,
from : result.follower,
to : result.user,
arrows: 'to, from'
});
}
})
}
else if (query.split(" ")[3] == "users") {
db.each(query, (err, result) => {
try{
nodes.add({
id : result.user,
label: result.user,
group: 'user'
});
}catch(err) {;}
})
}
else {
db.each(query, (err, result) => {
try{
nodes.add({
id : result.follows,
label: result.follows,
group: 'user'
});
}catch(err) {;}
try{
nodes.add({
id : result.user,
label: result.user,
group: 'user'
});
}catch(err) {;}
try{
edges.add({
id : result.user + "_" + result.follows,
from : result.user,
to : result.follows,
arrows: 'to'
});
}catch(err) {
edges.remove(result.user + "_" + result.follows);
edges.add({
id : result.user + "_" + result.follows,
from : result.user,
to : result.follows,
arrows: 'to, from'
});
}
})
}
query = "select * from ";
db.close();
};
function loadGraph(){
var container = document.getElementById('mynetwork');
data = {
nodes: nodes,
edges: edges
};
var options = {
layout: {improvedLayout:false},
physics : {
"barnesHut":{"damping": 1, "gravitationalConstant": -18500,},
stabilization: {
enabled : true,
iterations : 2000,
updateInterval: 25
},
timestep: 1,
},
groups: {
user: {shape: 'icon', icon:
{
face : 'Ionicons',
code : '\uf47e',
size : 50,
color: 'cyan'
}
}
},
nodes : {
font: {
color: '#ffffff'
}
}
};
network = new vis.Network(container, data, options);
network.on("stabilizationIterationsDone", function () {
network.setOptions( { physics: false } );
});
}
</script>
<div id="mynetwork"></div>
</body>
</html>
graph/pages/home.html deleted 100644 → 0
View file @ b74d7e46
<html>
<head>
<link href="node_modules/vis/dist/vis-network.min.css" rel="stylesheet" type="text/css"/>
<link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<link rel="stylesheet" href="http://code.ionicframework.com/ionicons/2.0.1/css/ionicons.min.css">
</head>
<body>
<h1>
Home of Twint OSINT Visualizer
</h1>
</body>
</html>
\ No newline at end of file
graph/photon/css/photon.min.css deleted 100644 → 0
View file @ b74d7e46
This diff is collapsed.
graph/photon/fonts/photon-entypo.svg deleted 100644 → 0
View file @ b74d7e46
This diff is collapsed.
graph/photon/template-app/app.js deleted 100644 → 0
View file @ b74d7e46
var app = require('app'); // Module to control application life.
var BrowserWindow = require('browser-window'); // Module to create native browser window.
// Keep a global reference of the window object, if you don't, the window will
// be closed automatically when the JavaScript object is garbage collected.
var mainWindow = null;
// Quit when all windows are closed.
app.on('window-all-closed', function() {
// On OS X it is common for applications and their menu bar
// to stay active until the user quits explicitly with Cmd + Q
if (process.platform != 'darwin') {
app.quit();
}
});
// This method will be called when Electron has finished
// initialization and is ready to create browser windows.
app.on('ready', function() {
// Create the browser window.
mainWindow = new BrowserWindow({
width: 600,
height: 300,
'min-width': 500,
'min-height': 200,
'accept-first-mouse': true,
'title-bar-style': 'hidden'
});
// and load the index.html of the app.
mainWindow.loadUrl('file://' + __dirname + '/index.html');
// Open the DevTools.
//mainWindow.openDevTools();
// Emitted when the window is closed.
mainWindow.on('closed', function() {
// Dereference the window object, usually you would store windows
// in an array if your app supports multi windows, this is the time
// when you should delete the corresponding element.
mainWindow = null;
});
});
graph/photon/template-app/index.html deleted 100644 → 0
View file @ b74d7e46
<!DOCTYPE html>
<html>
<head>
<title>Photon</title>
<!-- Stylesheets -->
<link rel="stylesheet" href="../css/photon.min.css">
<!-- Javascript -->
<script src="js/menu.js" charset="utf-8"></script>
</head>
<body>
<div class="window">
<!-- .toolbar-header sits at the top of your app -->
<header class="toolbar toolbar-header">
<h1 class="title">Photon</h1>
</header>
<!-- Your app's content goes inside .window-content -->
<div class="window-content">
<div class="pane-group">
<div class="pane pane-sm sidebar">
<nav class="nav-group">
<h5 class="nav-group-title">Favorites</h5>
<span class="nav-group-item">
<span class="icon icon-home"></span>
connors
</span>
<span class="nav-group-item active">
<span class="icon icon-light-up"></span>
Photon
</span>
<span class="nav-group-item">
<span class="icon icon-download"></span>
Downloads
</span>
<span class="nav-group-item">
<span class="icon icon-folder"></span>
Documents
</span>
<span class="nav-group-item">
<span class="icon icon-window"></span>
Applications
</span>
<span class="nav-group-item">
<span class="icon icon-signal"></span>
AirDrop
</span>
<span class="nav-group-item">
<span class="icon icon-monitor"></span>
Desktop
</span>
</nav>
</div>
<div class="pane">
<table class="table-striped">
<thead>
<tr>
<th>Name</th>
<th>Kind</th>
<th>Date Modified</th>
<th>Author</th>
</tr>
</thead>
<tbody>
<tr>
<td>bars.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>base.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>button-groups.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>buttons.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>docs.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>forms.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>grid.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>icons.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>images.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>lists.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>mixins.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>navs.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>normalize.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>photon.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>tables.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>tabs.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>utilities.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
<tr>
<td>variables.scss</td>
<td>Document</td>
<td>Oct 13, 2015</td>
<td>connors</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</body>
</html>
graph/photon/template-app/js/menu.js deleted 100644 → 0
View file @ b74d7e46
var remote = require('remote')
var Menu = remote.require('menu')
var MenuItem = remote.require('menu-item')
// Build our new menu
var menu = new Menu()
menu.append(new MenuItem({
label: 'Delete',
click: function() {
// Trigger an alert when menu item is clicked
alert('Deleted')
}
}))
menu.append(new MenuItem({
label: 'More Info...',
click: function() {
// Trigger an alert when menu item is clicked
alert('Here is more information')
}
}))
// Add the listener
document.addEventListener('DOMContentLoaded', function () {
document.querySelector('.js-context-menu').addEventListener('click', function (event) {
menu.popup(remote.getCurrentWindow());
})
})
graph/photon/template-app/package.json deleted 100644 → 0
View file @ b74d7e46
{
"name": "proton-template-app",
"version": "1.0.0",
"description": "A simple template app for Proton",
"main": "app.js",
"author": "Connor Sears",
"scripts": {
"start": "electron ."
}
}
twint/Pandas.py
View file @ 88a22f51
...@@ -24,7 +24,8 @@ def update(Tweet, session): ...@@ -24,7 +24,8 @@ def update(Tweet, session):
"link": Tweet.link, "link": Tweet.link,
"retweet": Tweet.retweet, "retweet": Tweet.retweet,
"user_rt": Tweet.user_rt, "user_rt": Tweet.user_rt,
"essid": str(session) "essid": str(session),
'mentions': Tweet.mentions
} }
_blocks.append(_data) _blocks.append(_data)
......
twint/config.py
View file @ 88a22f51
...@@ -44,4 +44,4 @@ class Config: ...@@ -44,4 +44,4 @@ class Config:
Search_name = "-" #for identify a records in mysql with the search it provides from. it cannot be null for DB requirements. a tweet must be in several search so the PK are tweet ID and search_name Search_name = "-" #for identify a records in mysql with the search it provides from. it cannot be null for DB requirements. a tweet must be in several search so the PK are tweet ID and search_name
Index_tweets = "twint" Index_tweets = "twint"
Index_follow = "twintGraph" Index_follow = "twintGraph"
Index_users = "twintUser" Index_users = "twintUser"
\ No newline at end of file
twint/db.py
View file @ 88a22f51
...@@ -198,7 +198,7 @@ def tweets(conn, Tweet, config): ...@@ -198,7 +198,7 @@ def tweets(conn, Tweet, config):
Tweet.user_rt, Tweet.user_rt,
",".join(Tweet.mentions), ",".join(Tweet.mentions),
date_time, date_time,
config.Search_name,) config.search_name,)
cursor.execute('INSERT INTO tweets VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)', entry) cursor.execute('INSERT INTO tweets VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)', entry)
conn.commit() conn.commit()
except sqlite3.IntegrityError: except sqlite3.IntegrityError:
......
twint/dbmysql.py
View file @ 88a22f51
...@@ -21,7 +21,7 @@ def init(hostname,Database,db_user,db_pwd): ...@@ -21,7 +21,7 @@ def init(hostname,Database,db_user,db_pwd):
passwd=db_pwd, # your password passwd=db_pwd, # your password
db=Database,# name of the data base db=Database,# name of the data base
charset='utf8mb4', charset='utf8mb4',
use_unicode=True) use_unicode=True)
cursor = conn.cursor() cursor = conn.cursor()
#here would be the code for creating the tables if them don't exist #here would be the code for creating the tables if them don't exist
return conn return conn
...@@ -33,7 +33,6 @@ def fTable(Followers): ...@@ -33,7 +33,6 @@ def fTable(Followers):
table = "followers_names" table = "followers_names"
else: else:
table = "following_names" table = "following_names"
return table return table
def uTable(Followers): def uTable(Followers):
...@@ -41,7 +40,6 @@ def uTable(Followers): ...@@ -41,7 +40,6 @@ def uTable(Followers):
table = "followers" table = "followers"
else: else:
table = "following" table = "following"
return table return table
def follow(conn, Username, Followers, User): def follow(conn, Username, Followers, User):
...@@ -74,13 +72,12 @@ def user(conn, Username, Followers, User): ...@@ -74,13 +72,12 @@ def user(conn, Username, Followers, User):
User.media_count, User.media_count,
User.is_private, User.is_private,
User.is_verified, User.is_verified,
User.avatar, User.avatar,
date_time, date_time,
Username,) Username,)
query = 'INSERT INTO {} VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'.format(uTable(Followers)) query = 'INSERT INTO {} VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'.format(uTable(Followers))
cursor.execute(query, entry) cursor.execute(query, entry)
conn.commit() conn.commit()
except MySQLdb.IntegrityError: except MySQLdb.IntegrityError:
pass pass
...@@ -106,7 +103,7 @@ def tweets(conn, Tweet, config): ...@@ -106,7 +103,7 @@ def tweets(conn, Tweet, config):
Tweet.user_rt, Tweet.user_rt,
",".join(Tweet.mentions), ",".join(Tweet.mentions),
date_time, date_time,
config.Search_name,) config.search_name,)
cursor.execute('INSERT INTO tweets VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)', entry) cursor.execute('INSERT INTO tweets VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)', entry)
conn.commit() conn.commit()
except MySQLdb.IntegrityError: except MySQLdb.IntegrityError:
......
twint/output.py
View file @ 88a22f51
...@@ -42,7 +42,10 @@ def _output(obj, output, config): ...@@ -42,7 +42,10 @@ def _output(obj, output, config):
if config.Store_object: if config.Store_object:
tweets_object.append(obj) tweets_object.append(obj)
else: else:
print(output) try:
print(output)
except UnicodeEncodeError:
pass
async def Tweets(tw, location, config, conn): async def Tweets(tw, location, config, conn):
copyright = tw.find("div", "StreamItemContent--withheld") copyright = tw.find("div", "StreamItemContent--withheld")
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment