New function to get dataset locations from MSPileup (addressing #11620)

Fixes #11620
Status
In development
Description
A new method is introduced to get the pileup dataset location leveraging MSPileup
Is it backward compatible (if not, which system it affects?)
NO (It's a new feature for fetching information)
Related PRs
None
External dependencies / deployment changes
No

From the second test of Jenkins test, I get this traceback:
Traceback (most recent call last):   File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py", line 298, in getDatasetLocationsFromMSPileup     doc = getPileupDocs(msPileupUrl, queryDict)   File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/MicroService/Tools/Common.py", line 119, in getPileupDocs     data = mgr.getdata(mspileupUrl, queryDict, headers, verb='POST',   File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/Services/pycurl_manager.py", line 363, in getdata     _, data = self.request(url=url, params=params, headers=headers, verb=verb,   File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/Utils/PortForward.py", line 66, in portMangle     return callFunc(callObj, newUrl, *args, **kwargs)   File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/Services/pycurl_manager.py", line 353, in request     raise exc http.client.HTTPException: url=https://cmsweb-prod.cern.ch:8443/ms-pileup/data/pileup, code=403, reason=Forbidden, headers={'Date': 'Tue, 30 Jan 2024 16:57:17 GMT', 'Server': 'Apache', 'Set-Cookie': 'cms-auth=3c656d3e60fea6a6a2d9956121c9b268bc31ac1f5ac876473d7677c5a14fc9678fc84ab3e0fc11d4;path=/;secure;httponly;expires=Thu, 01-Jan-1970 00:00:01 GMT', 'Content-Type': 'text/html;charset=utf-8', 'Content-Length': '753', 'X-Rest-Status': '100', 'X-Rest-Time': '1001.596 us', 'CMS-Server-Time': 'D=63250 t=1706633837604723'}, result=b'<!DOCTYPE html PUBLIC\n"-//W3C//DTD XHTML 1.0 Transitional//EN"\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\n    <title>403 Forbidden</title>\n    \n\n    \n
403 Forbidden
\n
You are not authorized to access this resource.
\n

\n
\n      \n        Powered by CherryPy 18.8.0\n      \n
\n    \n\n'
for WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase.testPileupData. Is this relevant?
@amaltaro

Dennis and I discussed the same issue yesterday. It turns out the query to MSPileup is incorrect. Before you make any further changes in this PR, I think it would be better to converge on Dennis' developments in https://github.com/dmwm/WMCore/pull/11870

@anpicci Andrea, now that we have the changes Dennis provided merged in master:
https://github.com/dmwm/WMCore/pull/11870
could you please update this PR accordingly? Please use the getPileupDocs() provided in the new python package. Thanks

Thank you @d-ylee @amaltaro. Do I need to rebase my local repo?

Yes @anpicci . Please let me know if you need any advice on that. If you prefer, you can also create a new development branch/PR.

@amaltaro from Jenkins logs, I read:
W0201, line 160 in StartPolicyInterface.call: Attribute 'mask' defined outside init
I have checked that actually it is the only self attribute defined outside the __init__, and the others are defined as None in the __init__ and modified afterwards. Should we apply the same strategy to self.mask?
In addition, I still get this error from the second test:
Traceback (most recent call last):
File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py", line 299, in getDatasetLocationsFromMSPileup
doc = getPileupDocs(msPileupUrl, queryDict, method='POST')[0]
File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/Services/MSPileup/MSPileupUtils.py", line 25, in getPileupDocs
data = mgr.getdata(mspileupUrl, queryDict, headers, verb=method,
File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/Services/pycurl_manager.py", line 363, in getdata
_, data = self.request(url=url, params=params, headers=headers, verb=verb,
File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/Utils/PortForward.py", line 66, in portMangle
return callFunc(callObj, newUrl, *args, **kwargs)
File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/3/label/cms-dmwm-cc7/code/src/python/WMCore/Services/pycurl_manager.py", line 353, in request
raise exc
http.client.HTTPException: url=https://cmsweb-prod.cern.ch:8443/ms-pileup/data/pileup, code=403, reason=Forbidden, headers={'Date': 'Thu, 01 Feb 2024 14:58:59 GMT', 'Server': 'Apache', 'Set-Cookie': 'cms-auth=575598d84b057ed202789f0472a4904a64268cff4b6685a92f03efd815dc2348a41d90116e365080;path=/;secure;httponly;expires=Thu, 01-Jan-1970 00:00:01 GMT', 'Content-Type': 'text/html;charset=utf-8', 'Content-Length': '753', 'X-Rest-Status': '100', 'X-Rest-Time': '777.721 us', 'CMS-Server-Time': 'D=98148 t=1706799539815054'}, result=b'\n\n\n    \n    403 Forbidden\n    \n\n    \n        403 Forbidden\n        You are not authorized to access this resource.\n        \n    \n      \n        Powered by CherryPy 18.8.0\n      \n    \n    \n\n'

@anpicci I suspect you didn't rebase this PR properly. You can rebase your development branch with the same instructions provided in this link: https://steveklabnik.com/writing/how-to-squash-commits-in-a-github-pull-request
just remove the -i option (for interactive squashing).

@amaltaro thanks to your correction, the error coming from MSPileup in the default test disappeared. Should I worry about the other errors in the default test?

I still see that this unit test is still failing:
    WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testPileupOnProduction changed from success to failure

While this one is likely unstable:
    WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testThrottling changed from success to failure

Giving it a second look, the first unit test is failing with:
root: ERROR: Error getting block location from MSPileup for /GammaGammaToEE_Elastic_Pt15_8TeV-lpair/Summer12-START53_V7C-v1/GEN-SIM: name 'dataItem' is not defined
Traceback (most recent call last):
  File "/build/cmsbld/jenkins/workspace/DMWM-WMCorePy3-PR-unittests/SLICE/8/label/cms-dmwm-cc7/code/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py", line 295, in getDatasetLocationsFromMSPileup
    queryDict = {'query': {'pileupName': dataItem},
NameError: name 'dataItem' is not defined

This error is reported in the middle of those 100s of log lines in this unit test: https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-WMCore-PR-test/14809/testReport/junit/WMCore_t.WorkQueue_t.WorkQueue_t/WorkQueueTest/testPileupOnProduction/

test please

With the last commit, the message in WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testPileupData
changed from:
root: INFO: pileupDatasets: {'https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader': {'/HighPileUp/Run2011A-v1/RAW'}}
root: INFO: datasetsWithDbsURL: [('https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader', '/HighPileUp/Run2011A-v1/RAW')]
root: INFO: dbsUrl: https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader
root: INFO: msPileupUrl: https://cmsweb-prod.cern.ch/ms-pileup/data/pileup
root: ERROR: Did not find any pileup document for query: {'pileupName': '/HighPileUp/Run2011A-v1/RAW'
to
root: INFO: pileupDatasets: {'https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader': {'/HighPileUp/Run2011A-v1/RAW'}}
root: INFO: datasetsWithDbsURL: [('https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader', '/HighPileUp/Run2011A-v1/RAW')]
root: INFO: dbsUrl: https://cmsweb-prod.cern.ch/dbs/prod/global/DBSReader
root: INFO: msPileupUrl: https://cmsweb-prod.cern.ch/ms-pileup/data/pileup
root: WARNING: Did not find any pileup document for query: {'pileupName': '/HighPileUp/Run2011A-v1/RAW'}
according to the modification I have made. This poses another question to me: Why do we get a 2011 file from here that is not present in the document returned by MSPileup, neither in prod nor in testbed? Is ignoring the pileup datasets from here that are not found in the MSPileup document correct?
In any case, the only error reported now as must be resolved is regarding WMCore_t.WorkQueue_t.WorkQueue_t/WorkQueueTest/testPileupOnProduction, but I am not sure how to fix it. I suspect it might relate to the missing pileup dataset.
@amaltaro

I see. In general, data location is resolved through Rucio, so far. And in many cases we are actually mocking calls to Rucio such that unit tests can reliable define the location for some of the DIDs that we use in the unit tests without actually making an HTTP request to a Rucio server.
I don't have an easy answer on how to mock that single MSPileup HTTP request. So we might have to change the pileup being used in that workqueue unit test to one of those that we actually have in MSPileup (we need to check if the workflow is using DBS - Rucio - testbed or production?). This might cause other issues with (un)mocked DBS data, but we will have to test that.

I see, is this a thing I can help with?

Yes, we have to adapt/fix that unit test according to the changes provided in here. If you are running unit tests over the docker image, you could adventure yourself on that code.
Otherwise I will try to pull your branch out and get that fixed, later in my day.

@anpicci Andrea, 2nd unit test can be fixed with the changes I provided in this file:
https://amaltaro.web.cern.ch/amaltaro/forAndreaP/Block_t.py
It is a sub-optimal fix though, given that it will only work inside this unit test module.
Ideally we should have a way to mock the module/function in https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/MSPileup/MSPileupUtils.py, but AFAIU, we would have to convert it to a class to have it properly emulated under: https://github.com/dmwm/WMCore/tree/master/src/python/WMQuality/Emulators
Please request another review once you have applied these modifications (and try to keep src/* changes separated from test/*).


Tiktok in mj-social-element

please add support for tiktok element in  mj-social-element

Same reply as here #2338 current social network are just here for compatibility with older MJML document. Just use https://documentation.mjml.io/#custom-social-element instead

oaky, thanks


Clarify that Trace Viewer not functional when server offline

When the server is offline, this information need to be communicated in a permanent and obvious way:

Critical information
Solution ideas

The tool cannot be used ...
Make the tool look and feel broken/unresponsive: Prevent interactions (ex. opening analyses), Fade out the associated UI (ex. Trace viewer side panel, trace tabs)

... because the trace server is offline
Write a more precise permanent message like "Trace Viewer Critical Error: Trace server offline"

Bonus: Point the user towards how they can fix the problem
Should they restart the server? Change the configuration?

Related: #53 comment, #234

Right now, when the trace server is offline the only hint to users is the bottom bar saying "Offline". PR #298 removed the server start/stop buttons from the bottom bar.

This is not clear enough considering that the Trace Viewer is unusable if the trace server is offline.

When the server is offline, it needs to be obvious to users that:
Neither of these things are clear in the current implementation:

The tool cannot be used - This is a critical failure
Right now, users can continue to attempt to add views (or interact with current views) which will never successfully update

Why the tool cannot be used - The trace server is offline
It's virtually impossible to guess that "Offline" means "The trace server is offline"

CC: @ssmagula

PR #298 removed the server start/stop buttons from the bottom bar.

The need for this was highlighted again in the user survey completed in November 2021.
Clarify when the trace server is offline:

This is critical information because the Trace Viewer does not work without the Trace Server
Users seem to not notice the bottom bar. Even if they did notice, they wouldn’t be able to guess that “Offline” means “The trace server is offline”.

@ebugden Wanted to get your opinion on an implementation we've been working on. The idea is to use polling to call the '/health' endpoint periodically (every 5 seconds for example), and make use of a status LED to indicate server status. The LED would indicate failure if the server is offline or when the '/health' endpoint returns 'status: "down"'. The LED has a tooltip to show more details when hovered:

We stop polling when the trace viewer side panel is detached, so that the ping is not constantly sent if the user decides to not continue with trace analysis

@hriday-panchasara Thank you for looking into this! @ssmagula could you review this design as I'm not actively contributing to the project now?


check current orientation

How to check current orientation in listener ?

Change orientation always in fullscreen mode . you can check activity orientation when fullscreen.


[SPARK-19327][SparkSQL] a better balance partition method for jdbc API

What changes were proposed in this pull request?
The partition method in jdbc using the equal
step, this can lead to skew between partitions.
The PR introduces a balance partition method base on the elements number when generate the partitions, this can relieve the skew problem with a little query cost.
How was this patch tested?
UnitTest and real data.

Can one of the admins verify this patch?

Here is the real data test result:
Table with 1.2Million rows, 50partition by SparkSQL.
old partition result(elements in each partition)

100061,100064,100059,100066,100065,100065,100066,100066,100063,100061,100066,100065,70747,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

new partition result(elements in each partition)

19543,19544,39083,39088,19544,19545,39085,19544,19542,19543,19545,39086,39087,19544,19545,39088,19544,19544,39088,19543,19545,39088,19544,19545,39088,19544,19544,39088,19544,19545,19543,19544,39086,19543,19545,39086,39086,19544,19545,39088,19544,19545,39088,19544,19544,39088,19544,19545,20701,0

count cost time: 1.27s

Table2 with about 5M rows, 200partition by SparkSQL.
(The table using the MySQL sharding, and every partition will return 10K rows at most)
old partition result(elements in each partition)

10000,49,54,53,60,59,48,61,52,57,60,69,58,57,50,52,51,66,58,45,59,52,61,56,67,51,45,49,70,49,58,59,61,53,50,53,47,50,46,53,55,53,62,55,48,58,52,62,62,37,65,59,58,55,61,59,46,53,49,49,61,72,60,46,50,51,45,47,55,63,64,63,55,47,65,57,60,60,51,45,48,77,58,57,59,39,50,62,55,57,49,63,51,38,49,66,62,58,53,54,50,54,52,69,51,49,61,60,64,49,52,50,54,58,48,51,50,49,41,68,54,45,65,62,44,52,64,58,47,51,65,47,37,42,39,44,51,65,56,54,69,51,61,63,51,52,47,55,58,66,47,54,53,53,60,66,66,68,64,66,55,58,64,55,50,57,46,56,39,60,57,63,40,51,56,58,44,46,46,44,42,52,52,44,53,46,55,57,68,57,62,48,47,52,59,58,49,44,52,47

(most of data is in partition 0, but each partition will return 10K at most because our sharding limit.)
new partition result(elements in each partition)

2083,10000,10000,6932,9799,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,8150,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,7,9,70,2,10000,10000,10000,655,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,40,76,145,38,86,176,369,696,1338,2776,5381'

count cost time: 0.8ms

@gatorsmile can you take a look at?

Yes, this solution is not suitable for large table, but I can not find a better solution, this is the best optimisation I can find.
So just add it as a choose, let the users know what he is doing, and need a explicit enable.
From my experience, the origin equal step method can lead to some problem for real data. This conclusion can be get from the spark-user email and our real scenario. Such as users will use the id to partition the table, because the id is unique and with index, but after many inserts and deletes, the id range is very large, and data will lead to a skew distribution by id.
Very large table is not so common, and if the large table with sharding, this method maybe acceptable.
My personal opinion is:

Given another choose for users maybe valuable, only we do not enable it by default.

Using the predicates parameters to split the table seems reasonable, but it just put some work should be done by Spark to users in my personal opinion. Users need know how to split the table uniform at first,  so it may use the count(*) extra to explode the distribution of the table.

So far, the best workaround is that predicate-based JDBC API; otherwise, as I mentioned above, we need to do it using sampling to find the boundary of each block.

In one embodiment, a user may specify a block size, via an interface. Blocks may be generated at the time of table partitioning. For example, according to a sampling technique described below, a user may select a particular block size and then the utility can determine the average number of table rows per block based on the number of storage bytes per row. Block-by boundary values for that range of rows of that block are determined based on the selected amount of rows, and provided in a query statement generated to obtain the statistical value for the block. That is, select rows from each table may be sampled or range-based. The select rows (or columns) are aggregated to form one “block” from the database table. The “block” may include the whole table, but is typically select rows of the whole table.

A few years ago, I did implement the sampling based table logical partitioning. See the link: https://www.google.com/patents/US20160275150. It works pretty well.

Yes, I agree with you, sampling bases is the right choose, but through jdbc API is not possible to achieve this.

It is still achievable if the SQL interface of the underlying database supports it. Currently, if the performance really matters, JDBC is not a good interface. Thus, most DBMS vendors provide their own Spark connectors.

Yes. I will leave this PR for a few days to seen if others interested in this, and then close it.

FWIW, I am negative of this approach too. It does not look a good solution to require full table scans to resolve skew between partitions.
As said, it is not good for a large table. Then, why don't we just repartition if the data is expected to be not quite large if we should resolve the skew?

@HyukjinKwon One assumption behind this design is that the specified column has index in most real scenario, so the table scan cost is not much high.
What I observed is that most large table has sharding, so count cost is acceptable, this is the reason
why we cost less time in a 5M rows table than in a 1M rows table. If we use the repartition,  there is a bottleneck  when loading data from DB and high cost for repartition.
Anyway, this solution is expensive indeed and not a good one, maybe the best way is using the Spark connectors provided by the DBMS vendors as @gatorsmile suggested.

The connectors by some DBMS vendors are using the UNLOAD utility, which performs much better, and build the RDD in the connectors.
Normally, JDBC is not a good option for large table fetching and writing.


 Syntax:

@CheckYerFlags star

Why would this even be helpful? Using bots to star is a known method of abuse and can lead to either suspension or termination of the bot account, and does have repercussions on the bot owner.

Yeah I think a chat mod Madara have suspend bots for this... my advice is to not do it

Ok, I have to row back here. It doesn't really help in anything. so consider the idea as discarded.


Typo in RFC for TFTP

In note 4 (https://docs.microsoft.com/en-us/sccm/core/plan-design/hierarchy/ports#bkmk_note4) the correct RFC for TFTP is 1350, not 350.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f658ee4b-d051-0d7c-4656-19a846908469
Version Independent ID: 3478e881-199c-f0c4-b218-0dd609eb0427
Content: Ports used for connections - Configuration Manager
Content Source: sccm/core/plan-design/hierarchy/ports.md
Product: configuration-manager
Technology: configmgr-other
GitHub Login: @aczechowski
Microsoft Alias: aaroncz

Thanks to @FrankRoj for fixing this issue. The change should publish live tomorrow (Tuesday, 14 May) around 11:00 Pacific time. I'm closing this issue.


refactor: custody withdraw args swap

swap token and recipient args in erc20 custody to keep compatibility with v1
did for all custody methods so that contract methods are consistent, no need to swap these args in other contracts imo, but if there is a need we can do it in other issue

@skosito another one that I think we could address in this PR: https://github.com/zeta-chain/protocol-contracts/issues/306

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.27%. Comparing base (6486dd8) to head (d4c04f5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #304   +/-   ##
=======================================
  Coverage   83.27%   83.27%
=======================================
  Files           7        7
  Lines         287      287
  Branches       95       95
=======================================
  Hits          239      239
  Misses         48       48

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update Firebase to 11.0

Hi @agusmarkosich8 ! In order to streamline the reporting and escalation process internally, we are disabling Github Issues for this repository. If you are still experiencing this issue please send a copy of your report, as well as any other information you might have, to<EMAIL_ADDRESS>so we can prioritize it accordingly. Thanks and apologies for the inconvenience!


Greenkeeper/@types/js yaml 3.11.4

Coverage remained the same at 96.296% when pulling 7e829c9a3c5c71673dc58fdd454df22f582047af on<EMAIL_ADDRESS>into 76d03dcab429a8f4c06c4ae8ae852cd028f0a194 on master.


Fix for Windows

I fixed two problems. The first is that the path separator is backslash on Windows so it will be outputted as a slash. The second fixes the problem of failing the test on Windows.
on command prompt
$ testbedder src\app\dashboard\dashboard.component.ts
? Which module do you use as real? DashboardComponent, └── HeroService
? Which module do you use as real? Done, DashboardComponent
import { DashboardComponent } from './dashboard.component';
import { HeroService } from '..\hero.service';
import { HeroServiceMock } from '..\hero.service.mock';

DashboardComponent,
{provide: HeroService, useClass: HeroServiceMock},

my environment:
OS:
Windows10
Node.js version:
6.10.0

Thanks!


[Feature] Add Verison Flag to CLI

containerssh issue 422
Added a 'version' variable to store version information
Implemented 'printVersion', 'runVersionCheck' function to print the current version using the '--version' flag

By contributing to this repository, I agree to the contribution guidelines.

If more flags are added, it seems like we'll need to split the file for management.
However, since this package operates in containers/pods, the usage of commands is minimal.
it' just an opinion.


The feature about feeding to discriminator

I find in the _train_discriminator() function in simplenet.py after the _embed() function, the feature becomes [10368, 1536], It's a two dimensions feature and no batch information, and then use this feature to train the discriminator. But for the output of the discriminator, it's like 10368 positive results and 10368 negetive result?  And no batch information. I got confused. Why they need do those.

The features of the batch are flattened(reshape) to 1D here.


Why Drizzle ORM and not Prisma DRM? (need support for MongoDB)

Hi - what is the reasoning behind the architecture design choice of picking Drizzle ORM instead of Prisma DRM?
I need to use MongoDB for my use case and from Drizzle ORM doesn't support no-SQL.
Regards,
-Avi

Hi, here is why the project choose Drizzle ORM instead of Prisma:
Schema in TypeScript
No DB client to generate
Programmatic migrate function
Config file.


init_globally throwing a run-time error

Expected Behavior
Should run properly
Actual Behavior
terminate called after throwing an instance of 'std::invalid_argument'
  what():  hpx::resource::get_partitioner() can be called only after the resource
           partitioner has been allowed to parse the command line options.
Aborted (core dumped)

This seems to happen only in release mode.
I had a very similar issue (either release or debug throwing that error, and the other not, but I don't remember if it was the same way around) trying to make an HPX-enabled library, which in the end was related to me not linking to HPX correctly. It may not be the same, but could be related (I will look into this next week).

FWIW, we use a similar global initialization scheme to enable HPX in a Python environment and have not seen any issues recently (see: https://github.com/STEllAR-GROUP/phylanx/blob/master/python/src/init_hpx.cpp)


Can't connect to Virtual Machine's server on Windows7 boot2docker 1.7.0

I can't make Docker run the Hello-world or any of server-connection things, as you can see it gives me the client version but at the time of giving me the server version it just can't connect...

I have boot2docker 1.7.0 and running on Windows 7.

As you can see here I dont get any error message at the start!

Sorry, this repository is long-since deprecated in favor of Docker Toolbox (whose usage is now also discouraged in favor of Docker for Windows and Docker for Mac).


Refactor: Break Apart Component Mutation Observer Logic

Jira
N/A
Summary
Slightly refactors + breaks out the mutation observer logic currently used in the button component into it's own standalone JS utility library for reuse in Bolt core.
How to test
Confirm button components rendering to the Shadow DOM re-render as expected when slotted content added (including the correct slot classes change accordingly).
Create a new button on the page and confirm the before / after works as expected.
const buttonIconBefore = document.createElement('bolt-icon');
buttonIconBefore.setAttribute('name', 'arrow-left');
buttonIconBefore.setAttribute('slot', 'before');

const buttonIconAfter = document.createElement('bolt-icon');
buttonIconAfter.setAttribute('name', 'chevron-right');
buttonIconAfter.setAttribute('slot', 'after');

const buttonElement = document.querySelector('bolt-button');
buttonElement.textContent = 'Hello world!';
buttonElement.appendChild(buttonIconAfter);

document.body.appendChild(buttonElement); // confirm the before slot has the `is-empty` CSS class.

buttonElement.appendChild(buttonIconBefore); // now confirm the before slot contains the new icon and no longer displays the `is-empty` CSS class.

Screenshot before the "before" slot is populated with content

Screenshot after the "before" slot is populated with content

Any idea why this PR didn't get a now.sh build @sghoweri?

Any idea why this PR didn't get a now.sh build @sghoweri?

@remydenton huh... that's weird!

@remydenton looks like this could either be:

a random now.sh issue OR

A random now.sh issue... possibly related to the number of concurrent builds / deploys we have going on? I'm curious to see if this problem goes away if we upgrade our now.sh plan to the next tier... particularly since I've seen this similar issue pop up a number of times now.

@remydenton working URL I pulled from the Travis logs: boltdesignsystem-vlrooyzhtk.now.sh


User-defined variables

Added new dialog to define user variables for a selected dataset. Whenever user variables change, they are stored in the browsers local storage given the selected dataset's identifier and the list of associated user variables.

A user variable can be selected as any other variable. In the list of dataset variables they are marked with a special icon.

User variables should work with all viewer functionality. With a user variable selected, check

info panel
time-series panel
statistics panel
volume panel

Closes #371
Requires backend implementation https://github.com/xcube-dev/xcube/pull/1021

What does min and max do? Is it the minimal and maximal values along the time axis? I have tried it out in the Viewer, but no data is shown.

What does min and max do? Is it the minimal and maximal values along the time axis? I have tried it out in the Viewer, but no data is shown.

True, I also the same question :)

Since this variables are user defined and will not be found in the data, should we disable Volume tab for user defined variables or is it that we are explicitly showing this message to user?

What does min and max do? Is it the minimal and maximal values along the time axis? I have tried it out in the Viewer, but no data is shown.

I have excluded min / max from the backend. Will do so for UI.

Since this variables are user defined and will not be found in the data, should we disable Volume tab for user defined variables or is it that we are explicitly showing this message to user?

That should be working now.


Revert "workaround to hide blue border box on ReportActionItem"

Details

Reverts Expensify/App#8443
Fixed Issues

$ https://github.com/Expensify/App/issues/12554
PROPOSAL: GH_LINK_ISSUE(COMMENT)
Tests

Open the app on safari web.
Sends a new message with some dummy text.

Add a numbered list of manual tests that can be performed by our QA engineers on the staging environment to validate that your changes work on all platforms, and that there are no regressions present.
Add any additional QA steps if test steps are unique to a particular platform.
Manual test steps should be written so that the QA engineer can repeat and verify one or more expected outcomes in the staging environment.

Select only a partial part of it via mouse.

Copy it using CMD+C.

Try to paste it into Composer.

Partially copied text should be pasted in the composer.

No change on Native platforms.

[x] Verify that no errors appear in the JS console

QA Steps
WEB | Desktop

Open the app on safari web.
Sends a new message with some dummy text.

Add a numbered list of manual tests that can be performed by our QA engineers on the staging environment to validate that your changes work on all platforms, and that there are no regressions present.
Add any additional QA steps if test steps are unique to a particular platform.
Manual test steps should be written so that the QA engineer can repeat and verify one or more expected outcomes in the staging environment.

Select only a partial part of it via mouse.
Copy it using CMD+C.
Try to paste it into Composer.
Partially copied text should be pasted in the composer.

Android | iOS

No change on Native platforms.

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] iOS / native
[x] Android / native
[x] iOS / Safari
[x] Android / Chrome
[x] MacOS / Chrome
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product was added in all src/languages/* files
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

PR Reviewer Checklist
The reviewer will copy/paste it into a new comment and complete it after the author checklist is completed

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots

Web

https://user-images.githubusercontent.com/24370807/201750424-95491589-e475-44ba-8fa5-efa57c0b55ba.mp4
Mobile Web - Chrome

Desktop

https://user-images.githubusercontent.com/24370807/201750832-060f2cd7-0e53-495c-84dd-0d88a2cccd8a.mp4
iOS

Android

This might be solving https://github.com/Expensify/App/issues/12010

@rushatgabhane think you'll be able to review this today? Should be an easy one.

I don't know how Rushat got assigned to this. There is no linking.

I think we automatically assign a different C+ than the creator of the PR. I went ahead and assigned @rushatgabhane to the issue as well so that we can get both of you paid out for this. @parasharrajat as the C (since you're raising the PR) and @rushatgabhane as the C+ (for reviewing). Cool?

Sounds good.

Checklists were filled out.

🚀 Deployed to staging by @puneetlath in version: 1.2.29-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅

@puneetlath @parasharrajat Can you pls confirm this is expected
it works with plain text but not with quoted text as in the PR.
https://user-images.githubusercontent.com/43995119/202742699-e285982e-84cc-4744-9c60-1c4a2c36b626.mp4


pin mpl to 3.4.1

the root of this issue has actually been resolved upstream in proplot, but the CI still isnt passing for some reason?

the issue is resolved with https://github.com/lukelbd/proplot/pull/268, and can be fixed on our end by pinning to the latest proplot once version 0.8.0 is released


Good news for Serenity community

I am really happy to see this.
Serene (Serenity Application Template) is now Highest Rated extension in https://marketplace.visualstudio.com/

Among all other extensions, Serene is the top rated.

Good Luck.
Thank you @volkanceylan for your hard work and patience.
This is the result.
May Allah bless you.

I'm happy to see that the meritocracy is still alive!

The community will keep rising i'm sure, if people help each other and spread the word :)

Old VSGallery was prioritizing extensions with more votes. So Serenity was in page 4-5 or so as it had less votes than others, even though rating was higher. Think this new one doesn't care much about number of votes like old one. Need more reviews to stay there, in case they change algorithm again.

Btw, we are still not among Top 200 projects with C# language in GitHub, sorted by stars. Have a long way to go.


Password Confirmation from documentation and the actual code does not match

Hi amazing package by the way really appreciate it.
I just wanted to notify that the confirmed rule format from the documentation still uses the {field}_confirmation format but when reviewing the code it uses the {field}_confirmed. Which confused me for a while when using the package.
Keep the solid work up 👌.

Hi @danieltejano!
Thank you very much for the feedback. ❤ I've fixed the documentation. 🚀


UpdateReply API and replyUpdateBlocker field for reply

As discussed in 20200212 meeting, we want to allow editor to edit their reply when

The reply is only used by the editor itself (the author should know the impact of the edit)
The reply has not yet received any feedback (the edit should now invalidate user's feedbacks)

We plan to add 1 mutation API, and one new field to Reply type for this:
"""
The reason why the editor cannot edit
"""
enum ReplyUpdateBlocker {
  NOT_AUTHORIZED # Not logged in or not the author of reply
  HAS_FEEDBACK
  USED_BY_OTHER_EDITORS
}

type Reply {
  """
  The reason why the current user cannot edit this reply. null if the current user can edit.

  The blockers are determined in the following order:
  (1) current user is the author or not
  (2) `positiveFeedbackCount` and `negativeFeedbackCount` are both 0 or not
  (3) all normal article-reply to this article are by the current author or not
  """
  updateBlocker: ReplyUpdateBlocker
}

mutation {
  """
  Updates reply text, type & reference when
  (1) current user is the author
  (2) `positiveFeedbackCount` and `negativeFeedbackCount` are both 0
  (3) all normal article-reply to this article are by the current author
  """
  UpdateReply(
    replyId: String!
    text: String
    type: ReplyTypeEnum
    reference: String
    waitForHyperlinks: Boolean = false
  ) {
    ""'
    The reply after update. `null` when update is not successful
    """
    reply: Reply

    """
    The reason why update fails. null when can update.
    """
    blocker: ReplyUpdateBlocker
  }
}

"all normal article-reply to this article are by the current author" can be achieved by loading article-reply of a reply using articleRepliesByReplyIdLoader and check each article-reply's author ID.

As discussed in https://g0v-slack-archive.g0v.ronny.tw/index/channel/C2PPMRQGP/2020-07#ts-1594556835.293700
The return type of the API can leverage union type, as Medium suggested.

As discussed in https://g0v.hackmd.io/OC7BneJ3TEGJHge1hWV-0w?both#討論：讓作者能編輯回應 the priority of this item is set to low.

Closed in favor of https://github.com/cofacts/rumors-site/issues/584


Cannot install nuget package

PM> Install-Package Microsoft.DotNet.ILCompiler -Version 1.0.0-alpha-26705-02 -Source https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
The 'Source' parameter is not respected for the transitive package management based project(s) IPBanProServiceInstaller. The enabled sources in your NuGet configuration will be used.
Restoring packages for XXXXX.csproj...
GET https://api.nuget.org/v3-flatcontainer/microsoft.dotnet.ilcompiler/index.json
NotFound https://api.nuget.org/v3-flatcontainer/microsoft.dotnet.ilcompiler/index.json 337ms
Install-Package : NU1101: Unable to find package Microsoft.DotNet.ILCompiler. No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages, nuget.org
At line:1 char:1

Install-Package Microsoft.DotNet.ILCompiler -Version 1.0.0-alpha-2670 ...

  + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
  + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

It seems like here, the package managed said it's going to ignore the Source parameter: The 'Source' parameter is not respected for the transitive package management based project(s) IPBanProServiceInstaller, and then it said it can't find the package in the sources you have preconfigured: No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages, nuget.org. Maybe adding MyGet to your sources in Visual Studio would help?


Fix E2E Memo test for broken versions

Summary of Bug
depends on #3026

The TestMsgTransfer_WithMemo is failing in certain configurations. See the following workflow to see the full failures.
The failing tests seem to all have this error:
RPC error -32603 - Internal error: tx (2F4F8F5D2FA006F9329153C4736CFEDC9F65924E6A5EC33E63065A310837D86D) not found

Expected Behaviour
The test should account for these versions and should pass.
Version
The following table shows which tests are broken.

Chain A
Chain B
Result

main
v4.1.1
PASS

main
v3.3.1
PASS

main
v2.4.2
PASS

main
v5.0.1
PASS

v4.1.1
main
FAIL

v3.3.1
main
FAIL

v2.4.2
main
FAIL

v5.0.1
main
FAIL

Steps to Reproduce
Run the TestMsgTransfer_WithMemo with any of these versions.

For Admin Use

[x] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

It looks like this is an issue with how the cosmos.BroadcastTx function works. We are discarding the tx response if the authtx query fails, which it will in the case of an unknown field. The test passing locally when updating the function in ibctest to
// BroadcastTx uses the provided Broadcaster to broadcast all the provided messages which will be signed
// by the User provided. The sdk.TxResponse and an error are returned.
func BroadcastTx(ctx context.Context, broadcaster *Broadcaster, broadcastingUser User, msgs ...sdk.Msg) (sdk.TxResponse, error) {
	f, err := broadcaster.GetFactory(ctx, broadcastingUser)
	if err != nil {
		return sdk.TxResponse{}, err
	}

	cc, err := broadcaster.GetClientContext(ctx, broadcastingUser)
	if err != nil {
		return sdk.TxResponse{}, err
	}

	if err := tx.BroadcastTx(cc, f, msgs...); err != nil {
		return sdk.TxResponse{}, err
	}

	txBytes, err := broadcaster.GetTxResponseBytes(ctx, broadcastingUser)
	if err != nil {
		return sdk.TxResponse{}, err
	}

	err = testutil.WaitForCondition(time.Second*30, time.Second*5, func() (bool, error) {
		var err error
		txBytes, err = broadcaster.GetTxResponseBytes(ctx, broadcastingUser)
		if err != nil {
			return false, nil
		}
		return true, nil
	})

	if err != nil {
		return sdk.TxResponse{}, err
	}

	respWithTxHash, err := broadcaster.UnmarshalTxResponseBytes(ctx, txBytes)
	if err != nil {
		return sdk.TxResponse{}, err
	}

	resp, err := authTx.QueryTx(cc, respWithTxHash.TxHash)
	if err != nil {
		// if we fail to query the tx, it means an error occurred with the original message broadcast.
		// we should return this instead.
		originalResp, err := broadcaster.UnmarshalTxResponseBytes(ctx, txBytes)
		if err != nil {
			return sdk.TxResponse{}, err
		}
		return originalResp, nil
	}
	return *resp, nil
}

We can wait for the sdk 47 PR to get merged in ibc test before making these changes. I don't think this is a big priority anymore as this confirms that the test still works, it was just an issue with the broadcast function.

I created a PR against the sdk 47 fork here


About Dataparallel

About all the hacky things that you wrote with comments, especially the things that have something to do with '(custom)dataparallel', is there anything that I have to know about its property when I want to add or remove some layers to the code? (multi GPU usage case)
Could you explain the following lines in detail for me ?
case1)
//A hack for the way dataparallel works
if cfg.share_prediction_module and pred_layer is not self.prediction_layers[0]:
    pred_layer.parent = [self.prediction_layers[0]]
=> parent double assignment on purpose?

(assignment already happened in PredictionModule constructor)
case2)
elif self.priors.device != device:
                # This whole weird situation is so that DataParalell doesn't copy the priors each iteration
                if prior_cache[size] is None:
                    prior_cache[size] = {}

                if device not in prior_cache[size]:
                    prior_cache[size][device] = self.priors.to(device)

                self.priors = prior_cache[size][device]

=> (whaaaat..?) device id consideration?
Well, I want to modify the code and retrain it with multiple GPUs but I'm afraid I could mess things up because of my lack of knowledge.

I think actually that comment was for a line of code I deleted earlier, whoops. The hack is actually for the way modules work, not dataparallel: modules count any module that's a member variable as part if its parameters and state_dict. However, I want to share the same "parent" module between multiple prediction heads (since FPN uses a shared prediction head). So to get around module's code, I cheekily put the parent in a python array which makes Module think it's not another module and thus doesn't add it to its state_dict. I don't think change is necessary there for additional layers, but keep that explanation in mind.

So the way dataparallel works is that it replicates the model on every GPU, computes the loss, then destroys the replicated copies. This makes sense if the things it's copying are weights that change for instance from one iteration to the next, but for priors which are fixed that's super inefficient. So I just have a cache for priors on each device so we don't copy them every iteration (also cross-device computation is a big no-no so we can't just have one copy on GPU 0).

To recap, a 'parent' should be wrapped with [] in order to share parameters between multiple prediction heads..?

One related, additional question.
Why do you assign parents two times?

PredictionModule constructor
yolact init
for idx, layer_idx in enumerate(self.selected_layers):
# If we're sharing prediction module weights, have every module's parent be the first one
parent = None
if cfg.share_prediction_module and idx > 0:
parent = self.prediction_layers[0]      <----- here
     pred = PredictionModule(src_channels[layer_idx], src_channels[layer_idx],
                             aspect_ratios = cfg.backbone.pred_aspect_ratios[idx],
                             scales        = cfg.backbone.pred_scales[idx],
                             parent        = parent,   **<----- here**
                             index         = idx)
     self.prediction_layers.append(pred)

PredictionModule init
self.parent      = [parent] # Don't include this in the state dict <----- here

yolact forward
for idx, pred_layer in zip(self.selected_layers, self.prediction_layers):
pred_x = outs[idx]
         if cfg.mask_type == mask_type.lincomb and cfg.mask_proto_prototypes_as_features:
             # Scale the prototypes down to the current prediction layer's size and add it as inputs
             proto_downsampled = F.interpolate(proto_downsampled, size=outs[idx].size()[2:], mode='bilinear', align_corners=False)
             pred_x = torch.cat([pred_x, proto_downsampled], dim=1)

         # A hack for the way dataparallel works
         if cfg.share_prediction_module and pred_layer is not self.prediction_layers[0]:
             pred_layer.parent = [self.prediction_layers[0]] **<----- here**

         p = pred_layer(pred_x)

         for k, v in p.items():
             pred_outs[k].append(v)

Oh... That's the reason why you used the global variable 'prior_cache' and save it for each device... I see..

Yup

Oh... I guess that's an artifact of when I didn't use CustomDataparallel. I'll try removing the assignment at (2) to see what it does.

Ok it looks like assigning it every iteration is necessary, because if we don't then the parent wont be on the same device as you're executing on. So it's really the first assignment that's unnecessary, because you definitely have to assign it every iteration. I think there's not much reason to change it now though.

I get it.
Thank you for your detailed explanation.
I learn many things from you!

No problem! I'll close the issue now.


WebAssembly is an 'internal' language module

Both the WebAssembly modules, wasm & wasm-wasi-component are internal modules. I.e they are both loaded by Unit as DSO's and run under the direct control of Unit in prototype/application processes.

Add the wasm and wasm-wasi-component modules to the modules directory listing.
$ git range-diff 96b5445...2470612
1:  96b5445 ! 1:  2470612 WebAssembly is an 'internal' language module
    @@ source/howto/modules.rst: In Node.js, Unit is supported by an :program:`npm`-hos
      .. _modules-emb:

      *************************
    +@@ source/howto/modules.rst: to find them in your system:
    +
    +    $ ls :nxt_ph:`/path/to/modules <Use runtime value if the default was overridden>`
    +
    +-         java.unit.so  perl.unit.so  php.unit.so  python.unit.so  ruby.unit.so
    ++         java.unit.so  php.unit.so     ruby.unit.so  wasm_wasi_component.unit.so
    ++         perl.unit.so  python.unit.so  wasm.unit.so
    +
    + To clarify the module versions, check the :ref:`Unit log <troubleshooting-log>`
    + to see which modules were loaded at startup:

Add Timo's Reviewed-by
$ git range-diff 2470612...fd3afb4
1:  2470612 ! 1:  fd3afb4 WebAssembly is an 'internal' language module
    @@ Commit message
         modules. I.e they are both loaded by Unit as DSO's and run under the
         direct control of Unit in prototype/application processes.

    +    Reviewed-by: Timo Stark<EMAIL_ADDRESS>         Signed-off-by: Andrew Clayton<EMAIL_ADDRESS>


What about nullable property templates?

Are they existing? If not pni or pin for public int? $propname$ { get; set; }
(I found your templates yesterday and already like them a lot :-) and I like the idea of those templates...)

                                                                                It's a possibility, also worth investigating if the ? character can be part of the template.                                                                                From: HopfenspirgerSent: Friday, 12 August 2016 09:17To: JetBrains/mnemonicsReply To: JetBrains/mnemonicsSubject: [JetBrains/mnemonics] What about nullable property templates? (#13)Are they existing? If not pni or pin for public int? $propname$ { get; set; }

(I found your templates yesterday and already like them a lot :-) and I like the idea of those templates...)
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/JetBrains/mnemonics","title":"JetBrains/mnemonics","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/JetBrains/mnemonics"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"What about nullable property templates? (#13)"}],"action":{"name":"View Issue","url":"https://github.com/JetBrains/mnemonics/issues/13"}}}

Yes, the ? can be part of the template. I just testet the following template and it works :-)
public long? $propname${ get; set; }$END$

What I mean is, whether or not the ? can be part of the shorthand used to
invoke the template, and whether having it as part of the template actually
autocompletes/expands as expected. Reason I'm curious is I remember
problems using the . (period) in the template shorthand - this didn't work
IIRC.
On 12 August 2016 at 15:33, Hopfenspirger<EMAIL_ADDRESS>wrote:

Yes, the ? can be part of the template. I just testet the following
template and it works :-)
public long? $propname${ get; set; }$END$
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JetBrains/mnemonics/issues/13#issuecomment-239462623,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcoNtfGyLM6WE5InDrHlC_lIqjvdltlks5qfIQpgaJpZM4Ji3jN
.

--
Dmitri Nesteruk
<EMAIL_ADDRESS>skype: dmitri.nesteruk

I just tried it. A dot ist not working because it is one of the "execute" signs for intellisense but a ? ist working. Still I would prefer n instead of ? for nullable because it is faster to type, at least on a german keyboard...


Update build process

Current setup.py is deprecated and produces deprecation warnings:
/root/xml2rfc/.tox/py39/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

/root/xml2rfc/.tox/py39/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.

We should update the build process in line with PEP 517 and PEP 518.

Duplicate of #781


feat(DatePicker): DatePicker implementation

Closes #379.
Hello,
First of all, I want to say thank you for this awesome library.
I implemented this feature taking inspiration from https://melt-ui.com/, https://www.bits-ui.com/docs/introduction and https://react-spectrum.adobe.com/react-stately, plus the examples provided on the issue.
The components that I've implemented are:

DateField
DateRangeField
Calendar
RangeCalendar
DatePicker
DateRangePicker

The last two are basically composing the field components, the calendar components and the popover component and are re-exporting them under the DatePicker/DateRangePicker naming.
The components are fully navigable by keyboard and should have all accessibility tags.
I open this as a draft PR because it still needs some stuff added to it.
The things that I did not manage to implement are: adding documentation, styling for the components in Histoire, usage examples, general cleanup and do some refactoring to clean the code and prevent unnecessary re-renders.
I appreciate any feedback and will implement any refactor, bugfix or feature required.

WHOAAAAA @epr3 !! I was just started looking into this, and you already have the PR ready!!! I'll review it shortly ya! 💚

@epr3 Thanks for your time ❤
Found a source that can guide you to add month or year selection, the project is also using @internationalized/date

https://www.marigold-ui.io/components/form/calendar#simple-calendar
https://github.com/marigold-ui/marigold/tree/main/packages/components/src/Calendar

I don't know which one is more convenient

Popover for selection
Overlay just like the demo sent above

@zernonia wdyt?

@sadeghbarati Thank you for your guidance.
I'm leaning towards implementing the year and month selection by providing two new primitives: CalendarYearSelect and CalendarMonthSelect which on click would either replace the values provided through the months v-slot value (and probably rename it to something more intuitive) by keeping an internal state for the view and letting the end-user re-use all the Calendar building blocks or providing separate values for dates, months and years and the current view through v-slot and letting the user the ability to make the decision on how he wants to display these values.
The only thing that I don't know how I should approach for this feature is what should the values be if number-of-months is set to something greater than 1 and paged-navigation is set to false.

Shadcn Svelte seems to have used a couple of selects for the month and year.
What do you think?

@epr3 It's nice that you create new Primitives for it ❤
And also use Vue features like slotProps 💚

The only thing that I don't know how I should approach for this feature is what should the values be if number-of-months is set to something greater than 1 and paged-navigation is set to false

Can you make a demo of this problem? You can push it in this PR so we could checkout to your branch and see what we can do about it

@sadeghbarati Unfortunately, I didn't start implementing this feature yet.
I want to think this edge case through.
For example, in the screenshot above, what months should be displayed on the Month Select View? (for one month it's: Jan, Feb, etc., for two months and paged navigation it would be: Jan-Feb, Mar-Apr, etc.), but if paged-navigation is set to false, the calendar would cycle like so: Jan-Feb, Feb-Mar, Mar-Apr, etc., therefore, the issue with the month display when you would click CalendarMonthSelect.

One possible solution that I can think of is to display the months like so: Jan, Feb, Mar, etc. and when you click one month it would set the start month on the calendar paged navigation.
Hope the description I provided makes sense.
Thanks!

Hello @zernonia and @sadeghbarati,
I marked this PR as ready for review.
Let me know if there's anything else that needs to be implemented or fixed.
Thank you!

@epr3 Again thanks for the time you put into this ❤

I see type error in radix-vue build script

The selected and focused date in Calendar and RangeCalendar will active the year cell background color, that green dot is enough I think, If I unfocus (not unselect) the selected date this black bg color will be removed

@epr3 Again thanks for the time you put into this ❤

I see type error in radix-vue build script
The selected and focused date in Calendar and RangeCalendar will active the year cell background color, that green dot is enough I think, If I unfocus (not unselect) the selected date this black bg color will be removed

I've fixed the typing bugs and removed the selected attribute on dates when the calendar is not in the month view.

What's the status of this? And is it going to be included in Shadcn-vue ?

What's the status of this? And is it going to be included in Shadcn-vue ?

Sure thing why not.

@epr3

Can we go next month by clicking those faded dates in the edge of Date Picker and Date Range Picker?

I think it's better to have same style as https://react-spectrum.adobe.com/react-aria/DateRangePicker.html for Range Pickers I mean data-selection-start="true" and data-selection-end="true"

Thanks @epr3 ! It's looking good! I will push some cleanup regarding the folder structure and stuff yaa 😁

@epr3  I see DateField, Calendar were broken if we use v-model in the story.

@zernonia - could you check the v-model with the latest commits?
From my testing, it seems that the ref kind of breaks when you use = to attribute one DateValue to another. Returning a new one works though.

yoo @epr3 ! I was looking through different case by adding more stories, and I noticed 2 bugs.

DateField is showing seconds if the set granularity="hour"
http://localhost:6006/story/packages-radix-vue-src-datefield-story-datefieldgranular-story-vue?variantId=packages-radix-vue-src-datefield-story-datefieldgranular-story-vue-2

Calendar setting preventSelect still enable user to deselect it .
http://localhost:6006/story/packages-radix-vue-src-calendar-story-calendarchromatic-story-vue?variantId=packages-radix-vue-src-calendar-story-calendarchromatic-story-vue-8

Hello @zernonia & @sadeghbarati,
Thank you very much for the help and reviews!
I have a dilemma regarding this point outlined by @sadeghbarati:

Layout shift in Calendar by clicking Heading Segment month and year which is not accessible I think using Overlay Component (Popover) is better to show months or years, one Popover which contains month and year logic

By using an Overlay Component(Popover), shouldn't the month and year selection feature, at this point, be handled through shadcn, by composing a normal Popover component?
I'm thinking that making the year and month selection an "opt-in" feature through the placeholder v-model field would reduce the amount of moving parts inside this component, the component itself would not need to re-export a Popover component(-4/5 components to worry about) and I think it would be more in line with the compositional nature of Radix.
I've created an example here (also removing the code pertaining to the month and year select):
https://github.com/epr3/radix-vue/blob/4f45fe931957079e07e483b49a54e01104ad10a9/packages/radix-vue/src/Calendar/story/CalendarPopover.story.vue
I'm leaning towards the year and month selection to be handled by shadcn, but I can also implement the Popover inside the Calendar and export all Popover components under a Calendar branding.
What do you guys think about this?

Sorry for causing a dilemma
I saw antd DatePicker
https://ant.design/components/date-picker, and those things come to my mind
Yeah, the opt-in feature is nice 🫡,
Instead of adding another component for it
With this, the user is not limited to using it only inside Popover.

Let's go with opt-in for Alpha version 😁

@epr3 why remove the defaultValue props?

They seem redundant, since the modelValue would get populated through the v-model.

@zernonia - I've added back the defaultValue props
I also would need some guidance on this issue if possible: Do you have any idea on how we could properly handle the DateValue values inside ref?
A lot of errors pop up due to the fact that Vue seems to drop nested values of DateValue (always returning a new DateValue doesn't always work), namely the calendar field which contains a lot of functions needed by @internationalized/date for various comparisons/operations and the package itself is built with ES6 classes which I believe builds Objects and populates their prototypes with class methods.

@epr3  unfortunately I dont think we can have a solution for now, as it seems like a problem with Vue.
https://www.reddit.com/r/vuejs/comments/xbfa3x/prop_as_a_typescript_class_object/
https://stackoverflow.com/questions/75557531/vue3-unwrap-properties-in-typescript-class
However we can get this PR merged. Absolutely banger job @epr3 !!!! 🤩

Nice! Looking forward to seeing this in Shadcn! @zernonia will you do a custom styling of the component to replace the old one, or as an entirely new one?

I would like to express my gratitude and emphasize the importance of component libraries that are consistently developed with accessibility in mind. I work for a service provider that creates websites for public entities (e.g., websites for municipalities and cities, which are legally required to be accessible!). We are legally obligated to implement guidelines that ensure accessibility. In Germany, these guidelines are known as BITV 2.0 ("Barrierefreie Informationstechnik-Verordnung 2.0"). Your framework and your work are incredibly important. Thank you!

Hello @epr3, how do you pass a DateTime instead of a Date? I need to control the time too.
I see that radix-vue exports a DateRange type but no DateTimeRange.
Could you please give me a hint on how to do this (with an example) ?
import type { DateRange } from 'radix-vue'

Congratulations on your work! It's fantastic!

Hello @valh1996,
You would need to use CalendarDateTime or ZonedDateTime  from the @internationalized/date package to be able to manipulate the time.
The DateRange type is meant to be used with the date range components: DateRangeField/DateRangePicker/RangeCalendar.

Hello @valh1996,
You would need to use CalendarDateTime or ZonedDateTime from the @internationalized/date package to be able to manipulate the time.
The DateRange type is meant to be used with the date range components: DateRangeField/DateRangePicker/RangeCalendar.

Ok, thanks, so if I understand correctly, the calendar doesn't support the time, so I have to manage it myself?

Yes, you would need to build the time picker yourself.

Will send a Stackblitz demo with description, but with a single calendar date


ConversationalRetrievalChain + Memory ( New issue for me)

Issue you'd like to raise.
I've tried building a Bot and now I see the following issue.
Architecture:

Cloud Run service 1 ; Streamlit app which accepts the user_input (question) on the topic and sends it to Flask API part of service 2.
Cloud Run service 2: Flask code which calls the LLM code and returns the response to streamlit.

I'm using qa as below.
    qa = ConversationalRetrievalChain.from_llm(
            llm,
            retriever,
            condense_question_prompt=CONDENSE_QUESTION_PROMPT,
            verbose=False,
            memory=memory,
            condense_question_llm =llm,
            return_generated_question= True,
            combine_docs_chain_kwargs={"prompt": promptHist},
            return_source_documents=True,
            )

Issue: If different Users are asking questions through various streamlit sessions, it is using the same chat_history buffer and mixing responses.
Question: Is there a way to separate the "qa" sessions specific to User and avoid common memory/chat buffer being used ? Is there a utility in Langchain for this?
Suggestion:
No response

Below is the sample code and trying to see  if it can be handled session wise.
def process_user_input(question, user_id, chat_history):
try:
print("Inside process input function")
    print("Parameters passed to LLM")
    print("User Input:", question)
    print("User ID:", user_id)
    print("Chat History:", chat_history)

    #memory = ConversationBufferMemory(memory_key="chat_history",input_key="question",output_key='answer',return_messages=True)
    qa = ConversationalRetrievalChain.from_llm(
            llm,
            retriever,
            condense_question_prompt=CONDENSE_QUESTION_PROMPT,
            verbose=False,
            memory=memory,
            condense_question_llm =llm,
            return_generated_question= True,
            combine_docs_chain_kwargs={"prompt": promptHist},
            return_source_documents=True,
            )

    print("QA chain")
    print(qa)
    print("memory:")
    print(memory)

    response = qa({"question": question, "chat_history": chat_history,"user_id": user_id})
    print(response)

@jimstechwork can you show the full code please? I've been trying to do something similar but I'm quite stuck and I'm not sure how to move on. I'm trying to use conversational retrieval chain along with a prompt and memory but that's not working.
My code part:
custom_template = ("""
Help the users with answering questions based on the document.
"""
)
# prompt = PromptTemplate(input_variables=[], template=custom_template)
prompt = PromptTemplate.from_template(custom_template)

memory = ConversationBufferMemory(memory_key = "chat_history", return_messages = True)
model = ConversationalRetrievalChain.from_llm(llm = llm,
                                               chain_type = "refine",
                                               retriever = new_knowledge_base.as_retriever(),
                                               memory = memory,
                                               combine_docs_chain_kwargs = {"prompt":prompt}
                                               )

Error:
Traceback (most recent call last):
File "C:\Users\Asus\Documents\Vendolista\app.py", line 100, in
main()
File "C:\Users\Asus\Documents\Vendolista\app.py", line 73, in main
model = ConversationalRetrievalChain.from_llm(llm = llm,
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\conversational_retrieval\base.py", line 341, in from_llm
doc_chain = load_qa_chain(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\question_answering_init_.py", line 249, in load_qa_chain
return loader_mapping[chain_type](
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\question_answering_init_.py", line 206, in _load_refine_chain
return RefineDocumentsChain(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\load\serializable.py", line 97, in init
super().init(**kwargs)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for RefineDocumentsChain
prompt
extra fields not permitted (type=value_error.extra)

@yazanrisheh - see if this helps.
p_template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
If the Standalone question is empty or cannot be generated, use the follow up question as Standalone question.
Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:"""
#CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)
CONDENSE_QUESTION_PROMPT = PromptTemplate(input_variables=["chat_history","question"],
template=p_template)
memory = ConversationBufferMemory(memory_key="chat_history",input_key="question",output_key='answer',return_messages=True)
chatTemplate = """
Important: You are an intelligent chatbot designed to help agents by answering questions only on Enterprise services & activities.
Answer the question only if there is information in the chat history(delimited by ) and context(delimited by  ) below.

If context is not empty and answer cannot be determined from context, say "I cannot detemine the answer from context".
If you don't know the answer, just say that you don't know, don't try to make up an answer.
Do not print your answer starting with "Answer:"

{context}

-----------

{chat_history}

-----------
Question: {question}
Answer:
Answer the question only if there is information based on the chat history(delimited by ) and context(delimited by  ) below.
1. If context is not empty and answer cannot be determined from context, say "I cannot detemine the answer from context".
2. If you don't know the answer, just say that you don't know, don't try to make up an answer.
3. Do not print your answer starting with "Answer:"
"""
promptHist = PromptTemplate(
input_variables=["context", "question", "chat_history"],
template=chatTemplate
)
qa = ConversationalRetrievalChain.from_llm(
llm,
retriever,
condense_question_prompt=CONDENSE_QUESTION_PROMPT,
verbose=False,
memory=memory,
condense_question_llm=llm,
return_generated_question=True,
combine_docs_chain_kwargs={"prompt": promptHist},
return_source_documents=True,
)

@jimstechwork I've been trying your code but it's still not working. I did some of my parts to it and this is the current code;
from dotenv import load_dotenv
import csv
import PyPDF2
from PyPDF2 import PdfReader
from langchain.document_loaders import DirectoryLoader, PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.callbacks import get_openai_callback
from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.prompts import PromptTemplate
import time
from langchain.vectorstores import Qdrant
from langchain.vectorstores import Chroma
from langchain.vectorstores import deeplake
from langchain.chains.qa_with_sources import load_qa_with_sources_chain
from langchain.callbacks import StreamingStdOutCallbackHandler
import pandas as pd
from docx import Document
from nltk.tokenize import sent_tokenize, word_tokenize
from collections import Counter
from nltk.corpus import stopwords
import os
def print_letter_by_letter(text):
for char in text:
print(char, end='', flush=True)
time.sleep(0.02)
def main():
load_dotenv()
my_activeloop_org_id = "yazanrisheh"
my_activeloop_dataset_name = "langchain_course_customer_support"
dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}"
# directory_path = input("Copy your directory path here or upload a file: ")
directory_path = "C:\Users\Asus\Documents\Vendolista"
pdf_loader = DirectoryLoader(directory_path,
                              glob="**/*.pdf",
                              show_progress=True,
                              use_multithreading=True,
                              silent_errors=True,
                              loader_cls = PyPDFLoader)

documents = pdf_loader.load()
print(str(len(documents))+ " documents loaded")

llm = ChatOpenAI(temperature = 0, model_name='gpt-3.5-turbo', callbacks=[StreamingStdOutCallbackHandler()], streaming = True)

# Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
)
chunks = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()

persist_directory = "C:\\Users\\Asus\\OneDrive\\Documents\\Vendolista"
knowledge_base = Chroma.from_documents(chunks, embeddings, persist_directory = persist_directory)
# save to disk
knowledge_base.persist()
#To delete the DB we created at first so that we can be sure that we will load from disk as fresh db
knowledge_base = None
new_knowledge_base = Chroma(persist_directory = persist_directory, embedding_function = embeddings)

# weird_knowledge_base = deeplake(chunks, dataset_path=dataset_path, embedding=embeddings)
# knowledge_base = Qdrant(documents, embeddings)

p_template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
If the Standalone question is empty or cannot be generated, use the follow up question as Standalone question.
Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:"""

#CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)
CONDENSE_QUESTION_PROMPT = PromptTemplate(input_variables=["chat_history","question"],
template=p_template)
memory = ConversationBufferMemory(memory_key="chat_history",input_key="question",output_key='answer',return_messages=True)

chatTemplate = """
Important: You are an intelligent chatbot designed to help agents by answering questions only on Enterprise services & activities.
Answer the question only if there is information in the chat history(delimited by ) and context(delimited by ) below.

If context is not empty and answer cannot be determined from context, say "I cannot detemine the answer from context".
If you don't know the answer, just say that you don't know, don't try to make up an answer.
Do not print your answer starting with "Answer:"
{context} ----------- {chat_history} ----------- Question: {question} Answer: Answer the question only if there is information based on the chat history(delimited by ) and context(delimited by ) below. 1. If context is not empty and answer cannot be determined from context, say "I cannot detemine the answer from context". 2. If you don't know the answer, just say that you don't know, don't try to make up an answer. 3. Do not print your answer starting with "Answer:"
"""

promptHist = PromptTemplate(
input_variables=["context", "question", "chat_history"],
template=chatTemplate
)

qa = ConversationalRetrievalChain.from_llm(
llm = llm,
retriever = new_knowledge_base,
condense_question_prompt=CONDENSE_QUESTION_PROMPT,
verbose=False,
memory=memory,
condense_question_llm=llm,
return_generated_question=True,
combine_docs_chain_kwargs={"prompt": promptHist},
return_source_documents=True,
)

while True:
    question = input("Ask me anything about the files (type 'exit' to quit): ")

    if question.lower() in ["exit"] and len(question) == 4:
        end_chat = "Thank you for visiting us! Have a nice day"
        print_letter_by_letter(end_chat)
        break

    if question:

        # chat_history = []
        # with get_openai_callback() as cb:
        response = qa({"question": question}, return_only_outputs = True)

        # chat_history.append(('user', question))
        # chat_history.append(('AI', response))
        print("AI:", response)

if name == 'main':
main()
The error:
Traceback (most recent call last):
File "C:\Users\Asus\Documents\Vendolista\app2.py", line 130, in
main()
File "C:\Users\Asus\Documents\Vendolista\app2.py", line 99, in main
qa = ConversationalRetrievalChain.from_llm(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\conversational_retrieval\base.py", line 356, in from_llm
return cls(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\load\serializable.py", line 75, in init
super().init(**kwargs)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for ConversationalRetrievalChain
retriever
value is not a valid dict (type=type_error.dict)
PS C:\Users\Asus\Documents\Vendolista> python app2.py
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:02<00:00, 11.17it/s]
87 documents loaded
Traceback (most recent call last):
File "C:\Users\Asus\Documents\Vendolista\app2.py", line 130, in
main()
File "C:\Users\Asus\Documents\Vendolista\app2.py", line 99, in main
qa = ConversationalRetrievalChain.from_llm(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\conversational_retrieval\base.py", line 356, in from_llm
return cls(
File "C:\Users\Asus\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\load\serializable.py", line 75, in init
super().init(**kwargs)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for ConversationalRetrievalChain
retriever
value is not a valid dict (type=type_error.dict)

@yazanrisheh - I used 2 templates to bring the customization aspect to the Conversational retrieval chain where you can feed in the customized template and try out.
Standalone question generation is required in the context of building a new question when an indirect follow-up question is asked in Chat

@jimstechwork I was able to work it out via 1 template but I have issues with the memory as its not remembering the past conversation. Could you tell me how to fix it? This is my current code:
from langchain.chat_models import ChatOpenAI
from langchain.prompts.prompt import PromptTemplate
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationalRetrievalChain
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Qdrant
from langchain.callbacks import StreamingStdOutCallbackHandler
from langchain.embeddings import OpenAIEmbeddings
from langchain.callbacks import get_openai_callback
from langchain.vectorstores import Chroma
from langchain.vectorstores import DeepLake
from PyPDF2 import PdfReader
import PyPDF2
from langchain.document_loaders import TextLoader, DirectoryLoader, UnstructuredWordDocumentLoader
from langchain.document_loaders import PyPDFDirectoryLoader, PyPDFLoader
from dotenv import load_dotenv
import time
Change DirectoryLoader since it has its own chunk size and overlap
load_dotenv()
directory_path = "C:\Users\Asus\Documents\Vendolista\MasterFormat"
pdf_loader = DirectoryLoader(directory_path,
glob="**/*.pdf",
show_progress=True,
use_multithreading=True,
silent_errors=True,
loader_cls = PyPDFLoader)
documents = pdf_loader.load()
print(str(len(documents))+ " documents loaded")
def print_letter_by_letter(text):
for char in text:
print(char, end='', flush=True)
time.sleep(0.02)
llm = ChatOpenAI(temperature = 0, model_name='gpt-3.5-turbo', callbacks=[StreamingStdOutCallbackHandler()], streaming = True)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=80,
)
chunks = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
persist_directory = "C:\Users\Asus\OneDrive\Documents\Vendolista"
knowledge_base = Chroma.from_documents(chunks, embeddings, persist_directory = persist_directory)
#save to disk
knowledge_base.persist()
#To delete the DB we created at first so that we can be sure that we will load from disk as fresh db
knowledge_base = None
new_knowledge_base = Chroma(persist_directory = persist_directory, embedding_function = embeddings)
prompt_template = """
You are a knowledgeable and ethical assistant with expertise in the construction field, including its
guidelines and operational procedures. You are always available to provide assistance and uphold the highest
ethical and moral standards in your work.
You will always and always and always only follow these set of rules and nothing else no matter what:

Answer the Question that is answering only based on the documents.
If the question is unrelated then say "sorry this question is completely not related.
If you think it is, email the staff and they will get back to you<EMAIL_ADDRESS>Do not ever answer with "I don't know" to any question no matter what kind of a question it is.
Everyime you answer a question, write on a new line "is there anything else you would like me to help you with?"
Remember to always take a deep breath and take this step by step.
Do not ever and under no circumstance do you rewrite the question asked by the customer when you are answering them.
If the question starts with the "Yazan," then you can ignore the steps above.

Text: {context}
Question: {question}
Answer :
"""
PROMPT = PromptTemplate(
template=prompt_template, input_variables=["context", "question"]
)
memory = ConversationBufferMemory(llm=llm, memory_key='chat_history', input_key='question', output_key='answer', return_messages=True)
conversation = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=new_knowledge_base.as_retriever(),
memory=memory,
chain_type="stuff",
verbose=False,
combine_docs_chain_kwargs={"prompt":PROMPT}
)
def main():
chat_history = []
while True:
    customer_input = input("Ask me anything about the files (type 'exit' to quit): ")

    if customer_input.lower() in ["exit"] and len(customer_input) == 4:
        end_chat = "Thank you for visiting us! Have a nice day"
        print_letter_by_letter(end_chat)
        break

    #if the customer_input is not equal to an empty string meaning they have input
    if customer_input != "":
        with get_openai_callback() as cb:
            response = conversation({"question": customer_input}, return_only_outputs = True)
        # print(response['answer'])
        print()
        print(cb)
        print()

if name == "main":
main()


Share verify_user_solution function

It's the exact same code copy pasted for all problem modules so might as well share it.
We might want some super fancy way to verify solutions for future problems but otherwise copy pasting code 20 times is probably bad.

Right now it seems we might want some problems in the future to have different verify_user_solution functions so closing. Also, there are more important things.


Alignment issues in Material-UI beta version

Description:
We are implementing APIM 3.0.0 UIs with Material - UI framework, in which there are alignment issues (Ex : with table pagination). We should upgrade it to newer version.
Suggested Labels:

Suggested Assignees:

Affected Product Version:
APIM 3.0.0
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

I’m closing this issue because it has been inactive for more than two years. Please reopen if this is still valid.


Why move away from home-manager?

I've been following your work since you posted those videos about nix flakes in your YouTube channel. It's really a great work, I even changed my nix dotfiles configuration to use nix flakes based in yours.
Now I see you are dropping home-manager entirely from your config, moving all to modules that maybe will require root password to be applied.
In my case this is something that I do not have in my Linux machine at work, which luckily I convinced IT personnel to install nix support in it, leaving me home-manager to configure my user directory without root privileges, which is great.
Do you have any strong reason to have done it?

Hi Rafael,
There is nothing wrong with home manager and I still recommend people use it. Your situation is actually a really good use case for it.
The reason I removed it from my config is I am trying to move to a setup where I have a small base system and then can layer flakes which have workspaces in them over the top.
I should probably break parts of my config out into another repo so its easier for other people to use.
Regards,
Wil
Sent from ProtonMail mobile
-------- Original Message --------
On 10 Nov 2021, 04:50, Rafael Brandão < @.***> wrote:

I've been following your work since you posted those videos about nix flakes in your YouTube channel. It's really a great work, I even changed my nix dotfiles configuration to use nix flakes based in yours.
Now I see you are dropping home-manager entirely from your config, moving all to modules that maybe will require root password to be applied.
In my case this is something that I do not have in my Linux machine at work, which luckily I convinced IT personnel to install nix support in it, leaving me home-manager to configure my user directory without root privileges, which is great.
Do you have any strong reason to have done it?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Hey, sorry, do you mind elaborating on this a bit? I watched your videos (which were great btw) to get an introduction to flakes... so I'm trying to move I guess, to a more modular system, where I can reuse configurations on different systems...
I've been using home-manager, but I'm just wondering about these workspaces....
Thanks!

@rembo10 Hey sorry about the delayed reply. The reason I decided to move to a workspace approach (term i made up I am working on a simple script to do it at the moment).
I found the home-manager and nixos system config approach didn't really work for my approach to using systems.

I have more machine specific things than I do user specific.
I found I had to make hacks to get machine specific settings into my home manager settings and the only reason I was using it was it made setting user specific settings easier.
Most of my computer use is actually based on different workflows. In a lot of cases I want to use the same programs but have separate configuration files and credentials etc.
I also find that having a docker image with all of my tools in it would be useful so I can run them when I am on other machines like work computers.

So I decided to make the following changes to my setup:

Move all of my settings into my system configuration so they are machine specific and not user specific. I made a small change to my system modules to make it easy to put user settings in.
Create my own workspace scripts so i can create a flake that has a set of tools in it and make it easy to jump into a shell with said tools and isolate the home directory of it so I can have separate settings if I want.
Once I have broken out all the software into workspace try to keep the system config as small as possible.

I am also going to create some templates that people can use too that still have various amounts of home manager integrated in them too since I don't think my own workflow will be all that useful to the majority of people.

@wiltaylor thanks for elaborating on that! I also just came from watching your youtube series (which was super helpful, thanks a lot for that!), and your use case matches mine pretty closely: different machines and VMs, both local and cloud; and different contexts on each machine (segmented by programming language; personal vs work stuff; different usernames and credentials for different github accounts, etc.) but I'm the only user on all of them.
So it seems like it would be easier to understand and manage a nixos config where I don't bring in an external dependency like home-manager that I don't understand in detail and probably won't use much of, and have everything be more explicit and visible to me.
I'm even considering at first just copy-pasting /etc/nixos/configuration.nix and /home/user/.config/nixpkgs/home.nix from your videos right into the flake.nix file and then delete the originals (also the later versions moved into .dotfiles) instead of importing them so I have a single file with everything inlined. Then as I learn more about nix and nixos I can start abstracting and extracting out and importing things that make sense.
Sort of like the elmlang people recommend you wait before a file gets uncomfortably large before trying to create directory structures and whatnot that might make things harder to move around and edit later.
Anyway NNTR and thanks for the great lessons and inspiration! Cheers.

@wiltaylor thanks so much for the thoughtful reply. I think I have a pretty similar setup so I'll be keeping an eye on the repo
I hope you keep making great videos!

I am also going to create some templates that people can use too that still have various amounts of home manager integrated in them too since I don't think my own workflow will be all that useful to the majority of people.

@wiltaylor hi thanks for your videos, very helpful.
It would be great that if you can elaborate your current setup with a blog post or video.
Cheers.

That would be nice. I'm planning out a migration to NixOS and your configs are very useful, but I am currently reluctant because I'm not comfortable enough changing (and breaking) things.

I know I'm sorry will try to get to it at some point soon. Just been snowed under with all kinds of stuff happening in my life for a bit.