-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathindex.html
More file actions
1631 lines (1485 loc) · 131 KB
/
Copy pathindex.html
File metadata and controls
1631 lines (1485 loc) · 131 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Q-Learning Simulation | Educational Platform</title>
<script src="https://cdn.tailwindcss.com"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
<!-- Load MathJax -->
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/contrib/auto-render.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css">
<link rel="stylesheet" href="asset/css/style.css">
<script>
tailwind.config = {
theme: {
extend: {
colors: {
primary: '#5D5CDE',
secondary: '#6D71F9',
bglight: '#F8F9FC',
cardlight: '#FFFFFF'
},
animation: {
'pulse-slow': 'pulse 3s cubic-bezier(0.4, 0, 0.6, 1) infinite',
'floating': 'float 3s ease-in-out infinite',
'bounce-slow': 'bounce 2s infinite',
},
keyframes: {
float: {
'0%, 100%': { transform: 'translateY(0)' },
'50%': { transform: 'translateY(-10px)' },
}
},
boxShadow: {
'glow': '0 0 15px 2px rgba(93, 92, 222, 0.3)',
'glow-lg': '0 0 25px 5px rgba(93, 92, 222, 0.4)',
'card': '0 10px 30px -5px rgba(0, 0, 0, 0.05)'
}
}
}
}
</script>
</head>
<body class="bg-bglight min-h-screen flex flex-col">
<div class="flex min-h-screen w-full relative">
<!-- Mobile sidebar toggle -->
<button id="sidebar-toggle" class="mobile-sidebar-toggle fixed top-4 left-4 z-50 bg-white p-3 rounded-full md:hidden shadow-lg border border-gray-100">
<i class="fas fa-bars"></i>
</button>
<!-- Sidebar -->
<div id="sidebar" class="sidebar w-72 bg-cardlight border-r border-gray-100 overflow-y-auto">
<div class="p-5 border-b border-gray-100 flex flex-col items-center">
<!-- Profile picture -->
<div class="w-24 h-24 rounded-full overflow-hidden border-4 border-primary shadow-lg mb-4 bg-white">
<img id="profile-photo" src="asset/img/me.png" alt="Abel Yohannes" class="w-full h-full object-cover">
</div>
<h1 class="text-xl font-bold text-gray-800 tech-font flex items-center">
<i class="fas fa-robot text-primary mr-2"></i>
Q-Learning Lab
</h1>
<p class="text-sm text-gray-600 mt-1">Interactive Education Platform</p>
</div>
<nav class="mt-4 px-3">
<div class="sidebar-menu-item active p-3 mx-2 cursor-pointer transition-all duration-200" data-section="home">
<div class="flex items-center">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-home text-primary"></i>
</div>
<span class="text-gray-700">Home</span>
</div>
</div>
<div class="sidebar-menu-item p-3 mx-2 cursor-pointer transition-all duration-200" data-section="about">
<div class="flex items-center">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-user text-primary"></i>
</div>
<span class="text-gray-700">About Me</span>
</div>
</div>
<div class="sidebar-menu-item p-3 mx-2 cursor-pointer transition-all duration-200" data-section="game">
<div class="flex items-center">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-gamepad text-primary"></i>
</div>
<span class="text-gray-700">Simulation</span>
</div>
</div>
<div class="sidebar-menu-item p-3 mx-2 cursor-pointer transition-all duration-200" data-section="theory">
<div class="flex items-center">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-book text-primary"></i>
</div>
<span class="text-gray-700">Q-Learning Theory</span>
</div>
</div>
<div class="sidebar-menu-item p-3 mx-2 cursor-pointer transition-all duration-200" data-section="qtable">
<div class="flex items-center">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-table text-primary"></i>
</div>
<span class="text-gray-700">Q-Table Visualization</span>
</div>
</div>
</nav>
<div class="mt-auto p-4 border-t border-gray-100">
<div class="text-xs text-gray-500">
<p>Version 1.1.0</p>
<p class="mt-1">© 2024 Abel Yohannes</p>
</div>
</div>
</div>
<!-- Main Content Area -->
<div class="content-area flex-1 overflow-y-auto">
<!-- Home Section -->
<section id="home" class="section-content p-4 md:p-8">
<div class="max-w-4xl mx-auto">
<div class="modern-card p-6 mb-6">
<div class="flex items-center mb-6">
<div class="mr-4 text-4xl text-primary">
<i class="fas fa-robot animate-bounce-slow"></i>
</div>
<div>
<h2 class="text-2xl font-bold text-gray-900 tech-font">Welcome to Q-Learning Lab</h2>
<div class="h-1 w-24 bg-gradient-to-r from-primary to-secondary rounded mt-1"></div>
</div>
</div>
<p class="text-gray-700 mb-6 leading-relaxed">
This interactive platform is designed to help you understand Reinforcement Learning, specifically
Q-learning and the Epsilon-Greedy algorithm through hands-on experimentation and visualization.
</p>
<div class="grid grid-cols-1 md:grid-cols-3 gap-6 mt-6">
<div class="hover-card p-5 flex flex-col">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-graduation-cap text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">
Learn by Doing
</h3>
<p class="text-gray-700 text-sm flex-1">
Interact with a live simulation to see how agents learn optimal paths through trial and error.
</p>
<div class="progress-indicator mt-4">
<div class="progress" style="width: 75%"></div>
</div>
</div>
<div class="hover-card p-5 flex flex-col">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-square-root-alt text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">
Mathematical Foundation
</h3>
<p class="text-gray-700 text-sm flex-1">
Understand the underlying mathematics from Bellman equations to Q-learning algorithms.
</p>
<div class="progress-indicator mt-4">
<div class="progress" style="width: 60%"></div>
</div>
</div>
<div class="hover-card p-5 flex flex-col">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-chart-line text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">
Visualize Learning
</h3>
<p class="text-gray-700 text-sm flex-1">
Watch Q-tables update in real-time as agents explore and learn from their environment.
</p>
<div class="progress-indicator mt-4">
<div class="progress" style="width: 90%"></div>
</div>
</div>
</div>
<div class="mt-8 bg-gradient-to-r from-primary/5 to-primary/10 rounded-xl p-6">
<div class="flex items-center mb-3">
<div class="w-10 h-10 rounded-lg bg-white border border-primary/20 flex items-center justify-center text-primary mr-4">
<i class="fas fa-rocket"></i>
</div>
<h3 class="text-lg font-semibold text-primary">Get Started</h3>
</div>
<p class="text-gray-700 mb-3">
Navigate through the sections using the sidebar to:
</p>
<div class="space-y-3 ml-2 mb-5">
<div class="flex items-center">
<div class="progress-circle active">1</div>
<div class="text-gray-700">Run simulations with different parameters</div>
</div>
<div class="flex items-center">
<div class="progress-circle active">2</div>
<div class="text-gray-700">Study the mathematical theory with interactive examples</div>
</div>
<div class="flex items-center">
<div class="progress-circle">3</div>
<div class="text-gray-700">Visualize Q-table updates in real-time</div>
</div>
<div class="flex items-center">
<div class="progress-circle">4</div>
<div class="text-gray-700">Explore different reinforcement learning concepts</div>
</div>
</div>
<div class="mt-5 flex justify-center">
<button id="goto-simulation-btn" class="animated-button bg-primary hover:bg-primary/90 text-white font-semibold py-3 px-8 rounded-lg transition flex items-center shadow-md">
<i class="fas fa-play mr-2"></i> Start Simulation
</button>
</div>
</div>
</div>
<div class="modern-card p-6">
<h2 class="text-xl font-bold text-gray-900 tech-font flex items-center mb-4">
<i class="fas fa-brain text-primary mr-3"></i>
<span>About Reinforcement Learning</span>
</h2>
<div class="h-1 w-32 bg-gradient-to-r from-primary to-secondary rounded mb-6"></div>
<p class="text-gray-700 mb-6 leading-relaxed">
Reinforcement Learning is a type of machine learning where an agent learns to make decisions
by taking actions in an environment to maximize cumulative reward. Unlike supervised learning,
the agent is not told which actions to take but must discover which actions yield the highest reward
through trial and error.
</p>
<div class="flex justify-center my-8">
<div class="relative">
<img src="asset/img/RL.jpg" alt="Reinforcement Learning Diagram"
class="max-w-full h-auto rounded-xl shadow-md border border-gray-100" style="max-height: 250px">
<div class="absolute -top-3 -right-3 bg-primary text-white text-xs font-bold px-3 py-1 rounded-full">
RL Process
</div>
</div>
</div>
<div class="clean-box">
<div class="flex items-center mb-3">
<div class="w-8 h-8 rounded-full bg-primary/10 flex items-center justify-center mr-3">
<i class="fas fa-lightbulb text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-gray-800">Q-learning in a Nutshell</h3>
</div>
<p class="text-gray-700 leading-relaxed">
Q-learning is a model-free reinforcement learning algorithm that learns the value of an action
in a particular state. It works by learning an action-value function (Q-function) that gives the expected
utility of taking a given action in a given state and following a fixed policy thereafter.
</p>
<div class="mt-5 grid grid-cols-1 md:grid-cols-3 gap-5">
<div class="bg-white p-4 rounded-lg text-center shadow-sm border border-gray-100">
<div class="text-primary text-xl mb-1">
<i class="fas fa-map-marker-alt"></i>
</div>
<div class="text-sm font-medium text-gray-700">States</div>
</div>
<div class="bg-white p-4 rounded-lg text-center shadow-sm border border-gray-100">
<div class="text-primary text-xl mb-1">
<i class="fas fa-arrows-alt"></i>
</div>
<div class="text-sm font-medium text-gray-700">Actions</div>
</div>
<div class="bg-white p-4 rounded-lg text-center shadow-sm border border-gray-100">
<div class="text-primary text-xl mb-1">
<i class="fas fa-gift"></i>
</div>
<div class="text-sm font-medium text-gray-700">Rewards</div>
</div>
</div>
<div style="font-family: 'Arial', sans-serif; font-size: 16px; color: #333; line-height: 1.6; background: linear-gradient(135deg, #f5f7fa, #c3cfe2); padding: 20px; border-radius: 10px; box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);">
<h2 style="font-size: 24px; color: #007BFF; margin-bottom: 15px; text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.1);">
Explore <strong style="color: #0056b3;">RL Tutorial</strong>
</h2>
<p style="margin-bottom: 20px;">
Dive into the world of Reinforcement Learning with our step-by-step guides and practical code examples. Learn about Q-learning, Deep Q-Networks (DQN), and more advanced techniques to master RL!
</p>
<a href="explain.html" style="display: inline-block; margin-top: 10px; padding: 12px 24px; background-color: #007BFF; color: #fff; text-decoration: none; border-radius: 25px; font-weight: bold; transition: background-color 0.3s ease, transform 0.2s ease; box-shadow: 0 4px 6px rgba(0, 123, 255, 0.2);">
Click here for Code Explainer
</a>
</div>
</div>
</div>
</div>
</section>
<!-- About Me Section -->
<section id="about" class="section-content p-4 md:p-8 hidden">
<div class="max-w-4xl mx-auto">
<div class="modern-card p-6 mb-6">
<div class="flex flex-col md:flex-row gap-6">
<div class="md:w-1/3">
<div class="rounded-xl overflow-hidden border-4 border-white shadow-md">
<img id="profile-photo-large" src="asset/img/me2.jpg" alt="Abel Yohannes" class="w-full h-auto">
</div>
<div class="mt-6 clean-box">
<h3 class="text-lg font-semibold text-gray-800 mb-3">Connect With Me</h3>
<div class="grid grid-cols-4 gap-3">
<a href="www.linkedin.com/in/abelyo252" class="flex items-center justify-center h-12 w-12 rounded-full bg-primary/10 hover:bg-primary/20 text-primary transition-all border border-primary/10">
<i class="fab fa-linkedin-in text-xl"></i>
</a>
<a href="https://github.com/abelyo252" class="flex items-center justify-center h-12 w-12 rounded-full bg-primary/10 hover:bg-primary/20 text-primary transition-all border border-primary/10">
<i class="fab fa-github text-xl"></i>
</a>
<a href="mailto:[email protected]" class="flex items-center justify-center h-12 w-12 rounded-full bg-primary/10 hover:bg-primary/20 text-primary transition-all border border-primary/10">
<i class="fas fa-envelope text-xl"></i>
</a>
</div>
</div>
</div>
<div class="md:w-2/3">
<h2 class="text-2xl font-bold text-gray-900 tech-font mb-2">Abel Yohannes</h2>
<div class="h-1 w-20 bg-gradient-to-r from-primary to-secondary rounded mb-4"></div>
<div class="flex flex-wrap gap-2 mb-4">
<span class="bg-primary/10 text-primary px-3 py-1 rounded-full text-sm font-medium">Computer Engineering</span>
<span class="bg-primary/10 text-primary px-3 py-1 rounded-full text-sm font-medium">AI Researcher</span>
<span class="bg-primary/10 text-primary px-3 py-1 rounded-full text-sm font-medium">Educator</span>
</div>
<p class="text-gray-700 mb-4 leading-relaxed">
I'm a Computer Engineering graduate from Jimma University (Class of 2023) with a passion
for Artificial Intelligence and a growing interest in financial systems. My research
focuses on reinforcement learning algorithms and their applications in real-world problems.
</p>
<p class="text-gray-700 mb-4 leading-relaxed">
I created this interactive Q-learning platform to help students and professionals
understand the foundational concepts of reinforcement learning through visual and
interactive examples. My goal is to make complex mathematical concepts more accessible
and intuitive.
</p>
<div class="mt-6 border-t border-gray-100 pt-6">
<h3 class="text-lg font-semibold text-gray-900 mb-4">My Expertise</h3>
<div class="space-y-4">
<div>
<div class="flex justify-between mb-1">
<span class="text-sm font-medium text-gray-700">Machine Learning</span>
<span class="text-sm text-primary font-semibold">95%</span>
</div>
<div class="w-full bg-gray-100 rounded-full h-2.5">
<div class="bg-gradient-to-r from-primary to-secondary h-2.5 rounded-full" style="width: 95%"></div>
</div>
</div>
<div>
<div class="flex justify-between mb-1">
<span class="text-sm font-medium text-gray-700">Reinforcement Learning</span>
<span class="text-sm text-primary font-semibold">90%</span>
</div>
<div class="w-full bg-gray-100 rounded-full h-2.5">
<div class="bg-gradient-to-r from-primary to-secondary h-2.5 rounded-full" style="width: 90%"></div>
</div>
</div>
<div>
<div class="flex justify-between mb-1">
<span class="text-sm font-medium text-gray-700">Web Development</span>
<span class="text-sm text-primary font-semibold">85%</span>
</div>
<div class="w-full bg-gray-100 rounded-full h-2.5">
<div class="bg-gradient-to-r from-primary to-secondary h-2.5 rounded-full" style="width: 85%"></div>
</div>
</div>
<div>
<div class="flex justify-between mb-1">
<span class="text-sm font-medium text-gray-700">Fintech Solutions</span>
<span class="text-sm text-primary font-semibold">80%</span>
</div>
<div class="w-full bg-gray-100 rounded-full h-2.5">
<div class="bg-gradient-to-r from-primary to-secondary h-2.5 rounded-full" style="width: 80%"></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="modern-card p-6">
<h2 class="text-xl font-bold text-gray-900 tech-font mb-2">Research Projects</h2>
<div class="h-1 w-20 bg-gradient-to-r from-primary to-secondary rounded mb-6"></div>
<div class="grid grid-cols-1 md:grid-cols-3 gap-6">
<div class="hover-card p-5 border border-gray-100">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-brain text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">Q-Learning in Dynamic Environments</h3>
<p class="text-gray-700 text-sm">
Research focused on how Q-learning algorithms can adapt to changing environment conditions
and maintain performance through adaptive learning rates and exploration strategies.
</p>
<div class="mt-4 flex justify-end">
<span class="text-xs font-medium py-1 px-2 bg-primary/10 text-primary rounded-full">2022-2023</span>
</div>
</div>
<div class="hover-card p-5 border border-gray-100">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-chart-line text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">Fintech Applications of RL</h3>
<p class="text-gray-700 text-sm">
Exploring how reinforcement learning can be applied to algorithmic trading and financial
decision-making systems to optimize performance under varying market conditions.
</p>
<div class="mt-4 flex justify-end">
<span class="text-xs font-medium py-1 px-2 bg-primary/10 text-primary rounded-full">2023-Present</span>
</div>
</div>
<div class="hover-card p-5 border border-gray-100">
<div class="w-12 h-12 rounded-lg bg-primary/10 flex items-center justify-center mb-4">
<i class="fas fa-graduation-cap text-xl text-primary"></i>
</div>
<h3 class="text-lg font-semibold text-primary mb-2">Educational Tools for AI Learning</h3>
<p class="text-gray-700 text-sm">
Development of interactive educational platforms that make complex AI concepts accessible
to students and professionals through visual simulations and guided learning paths.
</p>
<div class="mt-4 flex justify-end">
<span class="text-xs font-medium py-1 px-2 bg-primary/10 text-primary rounded-full">2023-Present</span>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Game Simulation Section -->
<section id="game" class="section-content p-4 md:p-8 hidden">
<div class="max-w-6xl mx-auto">
<div class="modern-card p-6 mb-6">
<div class="flex justify-between items-center mb-6">
<div>
<h2 class="text-xl font-bold text-gray-900 tech-font">Q-Learning Simulation</h2>
<div class="h-1 w-20 bg-gradient-to-r from-primary to-secondary rounded mt-1"></div>
</div>
<div class="flex gap-3">
<button id="reset-simulation-btn" class="animated-button bg-white border border-primary text-primary hover:bg-primary/5 text-sm py-2 px-4 rounded-lg transition flex items-center">
<i class="fas fa-sync-alt mr-2"></i> Reset
</button>
<button id="toggle-auto-btn" class="animated-button bg-primary hover:bg-primary/90 text-white text-sm py-2 px-4 rounded-lg transition flex items-center shadow-md">
<i class="fas fa-play mr-2"></i> Auto-Play
</button>
</div>
</div>
<div class="flex flex-col lg:flex-row gap-6">
<!-- Simulation Grid -->
<div class="lg:w-1/2">
<div class="clean-box">
<div class="flex justify-between items-center mb-5">
<h3 class="text-lg font-semibold text-gray-800 flex items-center">
<i class="fas fa-cube text-primary mr-2"></i>
Environment
</h3>
<div class="flex items-center bg-white py-1 px-3 rounded-full shadow-sm border border-gray-100">
<span class="text-sm text-gray-600">Episode: </span>
<span id="episode-counter" class="font-semibold text-primary ml-1">0</span>
</div>
</div>
<div id="simulation-container" class="relative aspect-square max-w-md mx-auto border border-gray-200 rounded-xl overflow-hidden shadow-md bg-white">
<div id="grid-container" class="grid grid-cols-5 grid-rows-5 w-full h-full">
<!-- Grid cells will be generated by JS -->
</div>
<div id="agent-element" class="absolute transition-all duration-300 flex items-center justify-center" style="width: 20%; height: 20%;">
<div class="agent">
<div class="agent-eye"></div>
<div class="agent-mouth"></div>
<div class="agent-shadow"></div>
</div>
</div>
<!-- Outcome screens -->
<div id="win-screen" class="absolute inset-0 bg-black/80 flex items-center justify-center flex-col hidden z-20">
<div class="text-6xl mb-4 animate-bounce-slow">🎉</div>
<h3 class="text-primary text-xl font-bold mb-3 tech-font">GOAL REACHED!</h3>
<p class="text-white text-center">
The agent found an optimal path!
</p>
</div>
<div id="fall-screen" class="absolute inset-0 bg-black/80 flex items-center justify-center flex-col hidden z-20">
<div class="text-6xl mb-4 animate-pulse-slow">💥</div>
<h3 class="text-red-500 text-xl font-bold mb-3 tech-font">AGENT FAILED</h3>
<p class="text-white text-center">
Learning from mistakes...
</p>
</div>
</div>
<div class="flex flex-wrap justify-center mt-6 gap-4">
<div class="flex items-center bg-white py-1 px-3 rounded-lg shadow-sm border border-gray-100">
<div class="w-4 h-4 safe-ice rounded-sm mr-2 border border-gray-200"></div>
<span class="text-sm text-gray-700">Safe</span>
</div>
<div class="flex items-center bg-white py-1 px-3 rounded-lg shadow-sm border border-gray-100">
<div class="w-4 h-4 hole rounded-sm mr-2"></div>
<span class="text-sm text-gray-700">Hole</span>
</div>
<div class="flex items-center bg-white py-1 px-3 rounded-lg shadow-sm border border-gray-100">
<div class="w-4 h-4 goal-cell rounded-sm mr-2"></div>
<span class="text-sm text-gray-700">Goal</span>
</div>
</div>
<div class="grid grid-cols-3 gap-2 mt-8">
<div></div>
<button id="move-up-btn" class="bg-white hover:bg-primary/5 text-gray-800 py-3 rounded-lg flex items-center justify-center shadow-sm transition-all border border-gray-100">
<i class="fas fa-chevron-up"></i>
</button>
<div></div>
<button id="move-left-btn" class="bg-white hover:bg-primary/5 text-gray-800 py-3 rounded-lg flex items-center justify-center shadow-sm transition-all border border-gray-100">
<i class="fas fa-chevron-left"></i>
</button>
<button id="move-down-btn" class="bg-white hover:bg-primary/5 text-gray-800 py-3 rounded-lg flex items-center justify-center shadow-sm transition-all border border-gray-100">
<i class="fas fa-chevron-down"></i>
</button>
<button id="move-right-btn" class="bg-white hover:bg-primary/5 text-gray-800 py-3 rounded-lg flex items-center justify-center shadow-sm transition-all border border-gray-100">
<i class="fas fa-chevron-right"></i>
</button>
</div>
</div>
</div>
<!-- Parameters & Live Q-Values -->
<div class="lg:w-1/2">
<div class="clean-box">
<h3 class="text-lg font-semibold text-gray-800 flex items-center mb-4">
<i class="fas fa-sliders-h text-primary mr-2"></i>
Learning Parameters
</h3>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4 mb-6">
<div class="bg-white p-4 rounded-lg shadow-sm border border-gray-100">
<label class="block text-sm font-medium text-gray-700 mb-2">
Learning Rate (α)
<span class="tooltip ml-1 text-gray-500">
<i class="fas fa-info-circle"></i>
<span class="tooltip-text">Controls how much new information overrides old information. Higher values make the agent learn faster.</span>
</span>
</label>
<div class="flex items-center">
<input type="range" id="learning-rate" min="0.1" max="1" step="0.1" value="0.7"
class="w-full h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer">
<span id="learning-rate-value" class="text-primary font-semibold ml-2 w-10 text-center">0.7</span>
</div>
</div>
<div class="bg-white p-4 rounded-lg shadow-sm border border-gray-100">
<label class="block text-sm font-medium text-gray-700 mb-2">
Discount Factor (γ)
<span class="tooltip ml-1 text-gray-500">
<i class="fas fa-info-circle"></i>
<span class="tooltip-text">Determines the importance of future rewards. Higher values make the agent consider long-term rewards more.</span>
</span>
</label>
<div class="flex items-center">
<input type="range" id="discount-factor" min="0.1" max="0.99" step="0.01" value="0.9"
class="w-full h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer">
<span id="discount-factor-value" class="text-primary font-semibold ml-2 w-10 text-center">0.9</span>
</div>
</div>
<div class="bg-white p-4 rounded-lg shadow-sm border border-gray-100">
<label class="block text-sm font-medium text-gray-700 mb-2">
Exploration Rate (ε)
<span class="tooltip ml-1 text-gray-500">
<i class="fas fa-info-circle"></i>
<span class="tooltip-text">Probability of taking a random action instead of the best known action. Higher values increase exploration.</span>
</span>
</label>
<div class="flex items-center">
<input type="range" id="exploration-rate" min="0.01" max="1" step="0.01" value="0.3"
class="w-full h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer">
<span id="exploration-rate-value" class="text-primary font-semibold ml-2 w-10 text-center">0.3</span>
</div>
</div>
<div class="bg-white p-4 rounded-lg shadow-sm border border-gray-100">
<label class="block text-sm font-medium text-gray-700 mb-2">
Exploration Decay
<span class="tooltip ml-1 text-gray-500">
<i class="fas fa-info-circle"></i>
<span class="tooltip-text">Rate at which exploration decreases over time. Higher values make the agent become more exploitative faster.</span>
</span>
</label>
<div class="flex items-center">
<input type="range" id="exploration-decay" min="0.9" max="0.999" step="0.001" value="0.98"
class="w-full h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer">
<span id="exploration-decay-value" class="text-primary font-semibold ml-2 w-10 text-center">0.98</span>
</div>
</div>
</div>
<h3 class="text-lg font-semibold text-gray-800 flex items-center mb-3">
<i class="fas fa-table text-primary mr-2"></i>
Live Q-Values
</h3>
<div class="bg-white p-4 rounded-lg shadow-sm mb-5 q-table-wrapper border border-gray-100">
<div class="flex justify-between items-center mb-3">
<div class="text-sm font-medium text-gray-700">
Q-Table (States × Actions): 25 × 4
</div>
<div class="flex gap-2">
<button id="show-full-qtable-btn" class="text-xs bg-primary/10 hover:bg-primary/20 text-primary py-1 px-2 rounded">
<i class="fas fa-expand-alt"></i> Full View
</button>
<button id="show-compact-qtable-btn" class="text-xs bg-gray-100 hover:bg-gray-200 text-gray-700 py-1 px-2 rounded">
<i class="fas fa-compress-alt"></i> Compact
</button>
</div>
</div>
<table class="min-w-full border-collapse q-table">
<thead>
<tr>
<th class="p-2 bg-gray-50 text-left text-xs font-medium text-gray-700 uppercase tracking-wider border border-gray-200 rounded-tl-lg">
State
</th>
<th class="p-2 bg-gray-50 text-left text-xs font-medium text-gray-700 uppercase tracking-wider border border-gray-200">
<i class="fas fa-arrow-up mr-1"></i> Up (0)
</th>
<th class="p-2 bg-gray-50 text-left text-xs font-medium text-gray-700 uppercase tracking-wider border border-gray-200">
<i class="fas fa-arrow-right mr-1"></i> Right (1)
</th>
<th class="p-2 bg-gray-50 text-left text-xs font-medium text-gray-700 uppercase tracking-wider border border-gray-200">
<i class="fas fa-arrow-down mr-1"></i> Down (2)
</th>
<th class="p-2 bg-gray-50 text-left text-xs font-medium text-gray-700 uppercase tracking-wider border border-gray-200 rounded-tr-lg">
<i class="fas fa-arrow-left mr-1"></i> Left (3)
</th>
</tr>
</thead>
<tbody id="q-table-body">
<!-- Q-values will be inserted here by JS -->
</tbody>
</table>
</div>
<div class="mb-5">
<h4 class="text-sm font-semibold text-gray-700 mb-2 flex items-center">
<i class="fas fa-code text-primary mr-2"></i>
Latest Q-Update
</h4>
<div id="q-update-formula" class="bg-white p-3 rounded-lg text-sm font-mono text-gray-700 overflow-x-auto shadow-sm border border-gray-100"></div>
</div>
<div>
<h4 class="text-sm font-semibold text-gray-700 mb-2 flex items-center">
<i class="fas fa-chart-bar text-primary mr-2"></i>
Statistics
</h4>
<div class="grid grid-cols-2 md:grid-cols-4 gap-3">
<div class="bg-white p-3 rounded-lg text-center shadow-sm hover:shadow transition-all border border-gray-100">
<div class="text-xs text-gray-500">Total Episodes</div>
<div id="stat-episodes" class="text-xl font-bold text-primary">0</div>
</div>
<div class="bg-white p-3 rounded-lg text-center shadow-sm hover:shadow transition-all border border-gray-100">
<div class="text-xs text-gray-500">Success Rate</div>
<div id="stat-success-rate" class="text-xl font-bold text-primary">0%</div>
</div>
<div class="bg-white p-3 rounded-lg text-center shadow-sm hover:shadow transition-all border border-gray-100">
<div class="text-xs text-gray-500">Avg. Steps</div>
<div id="stat-avg-steps" class="text-xl font-bold text-primary">0</div>
</div>
<div class="bg-white p-3 rounded-lg text-center shadow-sm hover:shadow transition-all border border-gray-100">
<div class="text-xs text-gray-500">Current ε</div>
<div id="stat-current-epsilon" class="text-xl font-bold text-primary">0.30</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Current Episode Details -->
<div class="modern-card p-6">
<h2 class="text-xl font-bold text-gray-900 tech-font mb-4 flex items-center">
<i class="fas fa-microscope text-primary mr-2"></i>
Episode Analysis
</h2>
<div class="h-1 w-20 bg-gradient-to-r from-primary to-secondary rounded mb-6"></div>
<div class="flex flex-col lg:flex-row gap-6">
<div class="lg:w-1/2">
<h3 class="text-lg font-semibold text-gray-800 mb-3 flex items-center">
<i class="fas fa-sitemap text-primary mr-2"></i>
Decision Process
</h3>
<div id="episode-log" class="bg-white p-4 rounded-lg h-64 overflow-y-auto text-sm text-gray-700 border border-gray-100 shadow-sm">
<div class="text-gray-500 italic">Episode log will appear here...</div>
</div>
</div>
<div class="lg:w-1/2">
<h3 class="text-lg font-semibold text-gray-800 mb-3 flex items-center">
<i class="fas fa-calculator text-primary mr-2"></i>
Q-Learning Steps
</h3>
<div id="q-learning-steps" class="bg-white p-4 rounded-lg h-64 overflow-y-auto text-sm text-gray-700 border border-gray-100 shadow-sm">
<div class="text-gray-500 italic">Q-learning calculations will appear here...</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Q-Learning Theory Section -->
<section id="theory" class="section-content p-4 md:p-8 hidden">
<div class="max-w-4xl mx-auto">
<div class="modern-card p-6 mb-6">
<h2 class="text-2xl font-bold text-gray-900 mb-2 tech-font">Q-Learning: Mathematical Foundation</h2>
<div class="h-1 w-32 bg-gradient-to-r from-primary to-secondary rounded mb-6"></div>
<div class="space-y-6">
<div class="clean-box bg-gradient-to-r from-primary/5 to-primary/10">
<h1 class="text-gray-800 text-center text-3xl font-semibold mb-4">Reinforcement Learning (RL) Explained</h1>
<p class="text-gray-700 leading-relaxed">
Reinforcement learning is an adaptive process in which an animal utilizes its previous experience to improve the outcomes of future choices. Computational theories of reinforcement learning play a central role in the newly emerging areas of neuroeconomics and decision neuroscience. In this framework, <strong>actions are chosen according to their value functions</strong>, which describe how much future reward is expected from each action. <strong> Value functions can be adjusted not only through reward and penalty, but also by the animal's knowledge of its current environment.</strong> Studies have revealed that a large proportion of the brain is involved in representing and updating value functions and using them to choose an action. However, how the nature of a behavioral task affects the neural mechanisms of reinforcement learning remains incompletely understood. Future studies should uncover the principles by which different computational elements of reinforcement learning are dynamically coordinated across the entire brain.
</p>
<div class="container text-center my-4">
<iframe width="560" height="315" src="https://www.youtube.com/embed/zl-MvcYQL0Q" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen class="mx-auto d-block"></iframe>
<h2>Here is some video for Mind Refereshing ...</h2>
</div>
<div class="mt-6">
<h2 class="text-gray-800 text-xl font-semibold mb-2">Model-free vs. model-based reinforcement learning</h2>
<p class="text-gray-700 leading-relaxed">
The brain's ability to learn from rewards and punishments, a process known as reinforcement learning (RL), can be elegantly described using mathematical frameworks. This article will guide you through the mathematical foundations of RL, starting with basic concepts and progressing to advanced techniques employed in modeling neural circuits.
</p>
<p class="text-gray-700 leading-relaxed">
In reinforcement learning, <strong>value functions</strong> help predict how rewarding an action or state will be. These functions can be updated in two main ways: through <strong>model-free</strong> and <strong>model-based</strong> learning. Let’s break this down using simple math.</p>
</p>
<h2 class="text-gray-800 text-xl font-semibold mb-1">1. Model-Free Learning: Updating with Reward Prediction Errors</h2>
<p class="text-gray-700 leading-relaxed">In <strong>model-free learning</strong>, value functions are updated based on the difference between the <strong>actual reward</strong> and the <strong>expected reward</strong>. This difference is called the <strong>reward prediction error (RPE)</strong>. In a class of reinforcement learning algorithms, referred to as simple or model-free reinforcement learning, reward prediction error is the primary source of changes in value functions. More specifically, the value function for the action chosen by the animal or the state visited by the animal is updated according to the reward prediction error, while the value functions for all other actions and states remain unchanged or simply decay passively</p><br>
<ul style="list-style-type: disc; padding-left: 20px;">
<li style="color: #4a5568; margin-bottom: 10px;">Let \( V(s) \) be the value of a state \( s \).</li>
<li style="color: #4a5568; margin-bottom: 10px;">Let \( r \) be the actual reward received.</li>
<li style="color: #4a5568; margin-bottom: 10px;">Let \( \gamma \) be a discount factor (how much we care about future rewards).</li>
<li style="color: #4a5568; margin-bottom: 10px;">Let \( V(s') \) be the value of the next state \( s' \).</li>
</ul>
<p style="color: #4a5568;">The reward prediction error (\( \delta \)) is calculated as:</p>
<p class="math">\[
\delta = r + \gamma V(s') - V(s)
\]</p>
<p class="text-gray-700 leading-relaxed">
This error tells us how wrong our prediction was. If \( \delta \) is positive, the reward was better than expected, so we increase \( V(s) \). If \( \delta \) is negative, the reward was worse, so we decrease \( V(s) \).
</p>
<p class="text-gray-700 leading-relaxed">The value function is updated as:</p>
<p class="math">\[
V(s) \leftarrow V(s) + \alpha \delta
\]</p>
<p class="text-gray-700 leading-relaxed">Here, \( \alpha \) is a learning rate (how quickly we update our predictions).</p><br>
<h2 class="text-gray-800 text-xl font-semibold mb-1">2. Model-Based Learning: Using Knowledge and Simulation</h2>
<p class="text-gray-700 leading-relaxed">In the second class of reinforcement learning, referred to as model-based reinforcement learning, value functions can be changed more flexibly. These algorithms can update the value functions on the basis of the animal’s motivational state and its knowledge of the environment without direct reward or penalty. The use of cognitive models allows the animal to adjust its value functions immediately, whenever it acquires a new piece of information about its internal state or external environment. There are many lines of evidence that animals as well as humans are capable of model-based reinforcement learning. For example, when an animal is satiated for a particular type of reward, the subjective value of the same food would be diminished. However, if the animal relies entirely on simple reinforcement learning, the tendency to choose a given action would not change until it experiences the devalued reward through the same action. Previous work has shown that rats can change their behaviors immediately according to their current motivational states following the devaluation of specific food items. This is often used as a test for goal-directed behaviors, and indicates that animals are indeed capable of model-based reinforcement learning. Humans and animals can also simulate the consequences of potential actions that they could have chosen. This is referred to as counterfactual thinking, and the information about hypothetical outcomes from unchosen actions can be incorporated into value functions when they are different from the outcomes predicted by the current value functions. Analogous to reward prediction error, the difference between hypothetical and predicted outcomes is referred to as fictive or counterfactual reward prediction error.</p><br>
<p class="text-gray-700 leading-relaxed">For example:</p>
<ul>
<li class="text-gray-700 leading-relaxed">If an animal is no longer hungry, it can <strong>immediately reduce</strong> the value of food-related actions, even without directly experiencing the devalued reward. This ability demonstrates how <strong>decision-making</strong> is not solely based on immediate feedback but also on <strong>internal states</strong> and <strong>learned associations</strong>. For instance, if a rat has learned that pressing a lever results in food, but it is no longer hungry, it can quickly devalue the action of pressing the lever because the reward (food) is no longer desirable. This process highlights the <strong>flexibility</strong> and <strong>adaptability</strong> of decision-making systems in both animals and humans.</li>
<li class="text-gray-700 leading-relaxed">Humans and animals can also simulate <strong>hypothetical outcomes</strong> of actions they didn’t take. This is called <strong>counterfactual thinking</strong>. Counterfactual thinking allows individuals to <strong>imagine alternative scenarios</strong> and outcomes, which can be used to evaluate past decisions or plan future actions. For example, if a person chooses to take a different route to work and gets stuck in traffic, they might imagine how much faster they would have arrived if they had taken their usual route. This <strong>mental simulation</strong> helps in learning from experiences and improving future decision-making.</li>
</ul>
<p class="text-gray-700 leading-relaxed">The difference between the <strong>hypothetical outcome</strong> (what could have happened) and the <strong>predicted outcome</strong> (what was expected to happen) is called the <strong>counterfactual reward prediction error</strong>. This concept works similarly to the standard <strong>reward prediction error</strong> but is based on <strong>imagined or counterfactual scenarios</strong> rather than actual experiences. For instance, if someone imagines that taking a different action would have led to a better outcome, the brain calculates the difference between this imagined outcome and the actual outcome. This <strong>counterfactual reward prediction error</strong> can influence future behavior by motivating individuals to adjust their strategies or actions to achieve better results in similar situations. It underscores the importance of <strong>mental simulation</strong> and <strong>imagination</strong> in learning and decision-making processes.</p>
</div>
<div class="mt-6">
<h2 class="text-gray-800 text-xl font-semibold mb-2">Conclusion</h2>
<p class="text-gray-700 leading-relaxed">
Reinforcement learning is a fundamental process that bridges neuroscience and evolutionary biology. It shapes behaviors that are crucial for survival, reproduction, and adaptation. The New Caledonian crow's ability to use tools and adapt through reinforcement learning highlights the intricate connection between behavior, brain mechanisms, and evolutionary success.
</p>
</div>
</div>
<div class="math-step">
<div class="flex items-start">
<div class="w-10 h-10 rounded-full bg-primary/10 flex items-center justify-center mt-1 mr-4 flex-shrink-0">
<i class="fas fa-dice-d6 text-primary"></i>
</div>
<div>
<h3 class="text-lg font-semibold text-primary mb-3">Markov Decision Process (MDP)</h3>
<p class="text-gray-700 mb-3">
In <strong>Reinforcement Learning</strong>, an agent operates in an environment modeled as a <strong>Markov Decision Process (MDP)</strong>. The MDP is defined as a 5-tuple \( \langle S, A, P, R, \gamma \rangle \), where:
</p>
<div class="formula-container" id="mdp-formula">
$$ \text{MDP} = \langle S, A, P, R, \gamma \rangle $$
</div>
<ul class="list-disc list-inside text-gray-700 mt-4 space-y-1">
<li><strong>S:</strong> Set of states \( s \in S \).</li>
<li><strong>A:</strong> Set of actions \( a \in A(s) \), where \( A(s) \) is the set of actions available at state \( s \).</li>
<li><strong>P:</strong> State transition probability \( P(s'|s, a) \), representing the probability of transitioning from state \( s \) to state \( s' \) upon action \( a \).</li>
<li><strong>R:</strong> Reward function \( R(s, a) \), assigning rewards to state-action pairs.</li>
<li><strong>\(\gamma\):</strong> Discount factor \( \gamma \in [0, 1] \), influencing the weight of future rewards.</li>
</ul>
<h4 class="text-xl font-semibold text-primary mt-6">1. Bellman Expectation Equation</h4>
<p class="text-gray-700">
The <strong>Bellman Expectation Equation</strong> computes the value of a state under a given policy \( \pi \), encapsulating the expected return when following that policy. Mathematically, it is expressed as:
</p>
<div class="formula-container">
$$ V^\pi(s) = R(s, \pi(s)) + \gamma \sum_{s'} P(s'|s, \pi(s)) V^\pi(s') $$
</div>
<p class="text-gray-700">
Alternatively, it can also be written as:
</p>
<div class="formula-container">
$$ V^\pi(s) = \sum_{a} \pi(a|s) \sum_{s'} P(s'|s, a) \left[ R(s, a, s') + \gamma V^\pi(s') \right] $$
</div>
<p class="text-gray-700">
<strong>Steps to calculate:</strong>
</p>
<ol class="list-decimal list-inside text-gray-700 mt-4 space-y-1">
<li><strong>Define the state space (S):</strong> List all the possible states in the environment.</li>
<li><strong>Define the policy \( \pi \):</strong> The policy \( \pi \) gives the probability of selecting each action \( a \) in each state \( s \).</li>
<li><strong>Define transition probabilities \( P(s'|s, a) \):</strong> This is the probability of transitioning from state \( s \) to state \( s' \) after taking action \( a \).</li>
<li><strong>Define the reward function \( R(s, a, s') \):</strong> This is the immediate reward received after taking action \( a \) in state \( s \) and transitioning to state \( s' \).</li>
<li><strong>Set the discount factor \( \gamma \):</strong> The discount factor controls the importance of future rewards.</li>
</ol>
<h1 class="text-xl font-semibold text-primary">The Bellman Equation: A Deep Dive</h1>
<p class="text-gray-700">
The <strong>Bellman Equation</strong> is one of the most fundamental concepts in <strong>Reinforcement Learning (RL)</strong>. It provides a recursive decomposition of the <strong>value function</strong>, which is used to evaluate how good it is for an agent to be in a particular state (or how good it is to take a particular action in a state). The Bellman Equation is the backbone of many RL algorithms, including <strong>value iteration</strong>, <strong>policy iteration</strong>, and <strong>Q-learning</strong>.
</p>
<p class="text-gray-700">
In this section, we'll break down the Bellman Equation in detail, starting from its intuition and building up to its mathematical formulation. We'll also explore its variants and how it is used in practice.
</p>
<h2 class="text-xl font-semibold text-primary">1. What is the Bellman Equation?</h2>
<p class="text-gray-700">
The Bellman Equation is a recursive equation that expresses the relationship between the value of a state (or state-action pair) and the values of its successor states. It is based on the principle of <strong>dynamic programming</strong>, which breaks down complex problems into simpler subproblems.
</p>
<p class="text-gray-700">
In RL, the Bellman Equation is used to compute the <strong>value function</strong> \( V(s) \) or the <strong>Q-function</strong> \( Q(s, a) \), which represent the expected cumulative reward an agent can achieve from a given state (or state-action pair).
</p>
<h4 class="text-xl font-semibold text-primary">Key Concepts</h4>
<p class="text-gray-700">
Before diving into the Bellman Equation, let's revisit some key concepts:
</p>
<h3 class="text-lg font-semibold text-primary">Value Function \( V(s) \):</h3>
<p class="text-gray-700">
The value function \( V(s) \) represents the expected cumulative reward an agent can achieve starting from state \( s \) and following a policy \( \pi \). It is defined as:
</p>
<div class="formula-container">
\[
V^\pi(s) = \mathbb{E}_\pi \left[ G_t \mid S_t = s \right]
\]
</div>
<p class="text-gray-700">Where:</p>
<ul class="text-gray-700">
<div class="formula-container">
<li>\( G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots \) is the <strong>return</strong> (cumulative discounted reward).</li>
<li>\( \gamma \) is the <strong>discount factor</strong> (between 0 and 1), which determines the importance of future rewards.</li>
</div>
</ul>
<h3 class="text-lg font-semibold text-primary">Q-Function \( Q(s, a) \):</h3>
<p class="text-gray-700">
The Q-function \( Q(s, a) \) represents the expected cumulative reward for taking action \( a \) in state \( s \) and then following policy \( \pi \). It is defined as:
</p>
<div class="formula-container">
\[
Q^\pi(s, a) = \mathbb{E}_\pi \left[ G_t \mid S_t = s, A_t = a \right]
\]
</div>
<h4 class="text-xl font-semibold text-primary"> Bellman Equation for the Value Function</h4>
<p class="text-gray-700">
The Bellman Equation for the value function \( V^\pi(s) \) expresses the value of a state \( s \) in terms of the values of its successor states \( s' \). It is derived from the recursive nature of the return \( G_t \):
</p>
<div class="formula-container">
\[
G_t = R_{t+1} + \gamma G_{t+1}
\]
</div>
<p class="text-gray-700">
Using this, we can write the value function as:
</p>
<div class="formula-container">
\[
V^\pi(s) = \mathbb{E}_\pi \left[ R_{t+1} + \gamma G_{t+1} \mid S_t = s \right]
\]
</div>
<p class="text-gray-700">
Expanding this expectation, we get:
</p>
<div class="formula-container">
\[
V^\pi(s) = \mathbb{E}_\pi \left[ R_{t+1} + \gamma V^\pi(S_{t+1}) \mid S_t = s \right]
\]
</div>
<p class="text-gray-700">
This is the <strong>Bellman Expectation Equation</strong> for the value function. It states that the value of a state \( s \) is the expected immediate reward plus the discounted value of the next state \( S_{t+1} \).
</p>
<h3 class="text-lg font-semibold text-primary">Mathematical Formulation:</h3>
<p class="text-gray-700">
For a given policy \( \pi \), the Bellman Equation for \( V^\pi(s) \) is:
</p>
<div class="formula-container">
\[
V^\pi(s) = \sum_{a} \pi(a|s) \sum_{s'} P(s' | s, a) \left[ R(s, a, s') + \gamma V^\pi(s') \right]
\]
</div>
<p class="text-gray-700">Where:</p>
<ul class="text-gray-700">
<li>\( \pi(a|s) \) is the probability of taking action \( a \) in state \( s \) under policy \( \pi \).</li>
<li>\( P(s' | s, a) \) is the transition probability of moving to state \( s' \) from state \( s \) after taking action \( a \).</li>
<li>\( R(s, a, s') \) is the reward received after transitioning from \( s \) to \( s' \) due to action \( a \).</li>
</ul>
<h4 class="text-xl font-semibold text-primary">Bellman Equation for the Q-Function</h4>
<p class="text-gray-700">
Similarly, the Bellman Equation for the Q-function \( Q^\pi(s, a) \) expresses the value of a state-action pair \( (s, a) \) in terms of the values of subsequent state-action pairs. It is given by:
</p>
<div class="formula-container">
\[
Q^\pi(s, a) = \mathbb{E}_\pi \left[ R_{t+1} + \gamma Q^\pi(S_{t+1}, A_{t+1}) \mid S_t = s, A_t = a \right]
\]
</div>
<p class="text-gray-700">
Expanding this expectation, we get:
</p>
<div class="formula-container">
\[
Q^\pi(s, a) = \sum_{s'} P(s' | s, a) \left[ R(s, a, s') + \gamma \sum_{a'} \pi(a'|s') Q^\pi(s', a') \right]
\]
</div>
<p class="text-gray-700">
This equation states that the value of taking action \( a \) in state \( s \) is the expected immediate reward plus the discounted value of the next state-action pair \( (s', a') \).
</p>
<h4 class="text-xl font-semibold text-primary">Bellman Optimality Equation</h4>
<p class="text-gray-700">
The <strong>Bellman Optimality Equation</strong> is a special case of the Bellman Equation that applies to the <strong>optimal value function</strong> \( V^*(s) \) and the <strong>optimal Q-function</strong> \( Q^*(s, a) \). These represent the maximum expected cumulative reward achievable from a state (or state-action pair) under the optimal policy \( \pi^* \).
</p>
<h3 class="text-lg font-semibold text-primary">Bellman Optimality Equation for \( V^*(s) \):</h3>
<div class="formula-container">
\[
V^*(s) = \max_{a} \sum_{s'} P(s' | s, a) \left[ R(s, a, s') + \gamma V^*(s') \right]
\]
</div>
<p class="text-gray-700">
This equation states that the optimal value of a state \( s \) is the maximum expected immediate reward plus the discounted optimal value of the next state \( s' \).
</p>
<h3 class="text-lg font-semibold text-primary">Bellman Optimality Equation for \( Q^*(s, a) \):</h3>
<div class="formula-container">
\[
Q^*(s, a) = \sum_{s'} P(s' | s, a) \left[ R(s, a, s') + \gamma \max_{a'} Q^*(s', a') \right]
\]