# Queue Configuration and Testing Guide

## Issues Fixed

1. ✅ Trade histories migration - Made open_time and close_time nullable
2. ✅ Created jobs table migration for database queue driver
3. ✅ Ran migrations successfully

## Manual Configuration Required

The .env file is protected by .gitignore. You need to manually configure the queue connection.

### Step 1: Edit .env File

Open `d:\Trade\FTP\.env` and verify/change this line:

```env
QUEUE_CONNECTION=database
```

**NOT** `QUEUE_CONNECTION=sync` (this runs jobs synchronously, which won't work for background processing)

### Step 2: Clear Configuration Cache

```bash
php artisan config:clear
```

### Step 3: Verify Queue Configuration

```bash
php artisan config:show queue
```

Should show:
```
queue.default => database
queue.connections.database.driver => database
queue.connections.database.table => jobs
```

## Why Timestamps Must Be Nullable in Trading Systems

### Trade Histories
- **open_time nullable**: Historical data imports may not have exact open times
- **close_time nullable**: Open positions don't have close times yet
- **Data integrity**: Better to have null than invalid default (e.g., 0000-00-00)

### Real-World Scenarios
1. **Data Migration**: Importing from legacy systems with incomplete data
2. **API Limitations**: Some broker APIs don't return all timestamps
3. **Partial Records**: Sync failures may leave incomplete records
4. **Testing**: Test data may not have realistic timestamps

## Queue Architecture for MT5 Sync

### Current Flow
```
Python → Laravel API → Dispatch Job → Database Queue → Queue Worker → Process
```

### Why Jobs Table is Required
When using `QUEUE_CONNECTION=database`, Laravel needs a table to store:
- Job class name
- Job payload (serialized data)
- Queue name
- Attempts count
- Reserved until timestamp
- Available at timestamp
- Created at timestamp

### Jobs Table Structure
```sql
CREATE TABLE `jobs` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `queue` varchar(255) NOT NULL,
  `payload` longtext NOT NULL,
  `attempts` tinyint unsigned NOT NULL,
  `reserved_at` int unsigned DEFAULT NULL,
  `available_at` int unsigned NOT NULL,
  `created_at` int unsigned NOT NULL,
  PRIMARY KEY (`id`),
  KEY `jobs_queue_index` (`queue`)
)
```

## Testing Queue Worker

### Terminal 1: Start Laravel Server
```bash
cd d:/Trade/FTP
php artisan serve
```

### Terminal 2: Start Queue Worker
```bash
cd d:/Trade/FTP
php artisan queue:work
```

Expected output:
```
   INFO  Processing jobs from the [default] queue.
   INFO  Processing: App\Jobs\ProcessMt5Sync
   INFO  Processed:  App\Jobs\ProcessMt5Sync
```

### Terminal 3: Run Python Sync
```bash
cd python-mt5-bridge
python mt5_sync.py
```

Expected output:
```
INFO - MT5 sync job queued successfully
```

## Troubleshooting Checklist

### Queue Configuration
- [ ] QUEUE_CONNECTION=database in .env
- [ ] jobs table exists in database
- [ ] php artisan config:clear run
- [ ] php artisan config:show queue shows correct values

### Queue Worker
- [ ] Queue worker is running (php artisan queue:work)
- [ ] Worker can connect to database
- [ ] Worker has permission to write to jobs table
- [ ] Worker is processing jobs from correct queue

### Job Dispatching
- [ ] Job is being dispatched in controller
- [ ] Job payload is valid
- [ ] Job class exists and is importable
- [ ] Job handle() method has no syntax errors

### Database
- [ ] jobs table exists
- [ ] Database connection is working
- [ ] User has INSERT/UPDATE/DELETE permissions on jobs table
- [ ] MySQL is running

## Production Queue Architecture

### Supervisor Configuration (Linux)
```ini
[program:ecm-queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/ecm-platform/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/ecm-platform/storage/logs/worker.log
stopwaitsecs=3600
```

### Windows Service (NSSM)
```powershell
nssm install ECMQueueWorker "C:\php\php.exe" "C:\path\to\artisan queue:work"
nssm set ECMQueueWorker AppDirectory "C:\path\to\project"
nssm start ECMQueueWorker
```

### Recommended Queue Settings
```bash
php artisan queue:work \
  --sleep=3 \        # Wait 3 seconds between jobs
  --tries=3 \        # Retry failed jobs 3 times
  --timeout=3600 \   # Max 1 hour per job
  --max-jobs=1000 \  # Process 1000 jobs then restart
  --max-time=3600    # Restart worker every hour
```

## Monitoring Queue Jobs

### List Failed Jobs
```bash
php artisan queue:failed
```

### Retry All Failed Jobs
```bash
php artisan queue:retry all
```

### Retry Specific Failed Job
```bash
php artisan queue:retry <job-id>
```

### Flush Failed Jobs
```bash
php artisan queue:flush
```

### Clear All Jobs
```bash
php artisan queue:clear
```

## Next Steps

1. Edit .env to set QUEUE_CONNECTION=database
2. Run: php artisan config:clear
3. Start queue worker: php artisan queue:work
4. Test Python sync: python mt5_sync.py
5. Check Laravel logs: tail -f storage/logs/laravel.log