The Geographically Weighted Regression (GWR) python code fails to run in the Jupyterlab of Anaconda,

The Geographically Weighted Regression (GWR) python code fails to run in the Jupyterlab of Anaconda,
I couldn’t figure out why the Geographically Weighted Regression (GWR) python code fails to run in the Jupyterlab of Anaconda and ends up with an error,
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 3
1 import pandas as pd
2 import numpy as np
----> 3 from pysal.spreg import GWR
4 from pysal.lib.weights import DistanceBand
6 # Read the data from the Excel file
ModuleNotFoundError: No module named 'pysal.spreg'

GWR lives in its own sub module rather than in spreg. Try from
pysal.mgwr.gwr import GWR, MGWR
On Wed, Nov 8, 2023 at 5:07 PM jamalnuman68108 @.***>
wrote:

The Geographically Weighted Regression (GWR) python code fails to run in
the Jupyterlab of Anaconda,
I couldn’t figure out why the Geographically Weighted Regression (GWR)
python code fails to run in the Jupyterlab of Anaconda and ends up with an
error,
ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 3
1 import pandas as pd
2 import numpy as np
----> 3 from pysal.spreg import GWR
4 from pysal.lib.weights import DistanceBand
6 # Read the data from the Excel file
ModuleNotFoundError: No module named 'pysal.spreg'
[image: Clip_504]
https://user-images.githubusercontent.com/150298816/281556319-90b0ca78-3a59-4f85-826b-b0ee3896ffb9.jpg
[image: Clip_505]
https://user-images.githubusercontent.com/150298816/281556344-70dcd852-388a-4484-8c8c-56527128e0cf.jpg
—
Reply to this email directly, view it on GitHub
https://github.com/pysal/pysal/issues/1287, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB553TPWFLJXOCWV6QDOFTTYDP7DRAVCNFSM6AAAAAA7DS3P6CVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE4DINBXGMYDAMY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

both ChatGPT and Google Bard fail to get the code below:

Write a Python (version 3.12.0) code to read the data from the table below
"C:\D\R1.xlsx"
then
Write a Python (version 3.12.0) code for GWR (Geographically Weighted Regression) to predict the value of the dependent variable based on the independent variables.
Use the JupyterLab of the Anaconda
use the pysal.mgwr.gwr import GWR, MGWR
Assumptions:

The first column is the dependent variable
All other columns are the independent variables
The last two columns which are u and v represent the X and Y coordinates, respectively, of the location of the observation
The bandwidth is 500 meters (fixed distance)
Use Gaussian as a kernel function

Find:

The rmse, mae, mape, and r2 to evaluate the model
Display the predicted field with the data
Display a field that represents the difference between the actual value and the predicted value (residual)

according to chatGPT:

import pandas as pd
import geopandas as gpd
from pysal.lib import weights
from pysal.explore import mgwr
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
Step 1: Read the data from the Excel file
file_path = "C:\D\R1.xlsx"
data = pd.read_excel(file_path)
Step 2: Define dependent and independent variables
dependent_variable = data.iloc[:, 0]  # Assuming the first column is the dependent variable
independent_variables = data.iloc[:, 1:-2]  # All columns except the last two
Step 3: Create a GeoDataFrame with coordinates
x_coords = data['u'].values
y_coords = data['v'].values
gdf = gpd.GeoDataFrame(data, geometry=gpd.points_from_xy(x_coords, y_coords))
Step 4: Create a spatial weights matrix
w = weights.distance.Kernel.from_dataframe(gdf, fixed=True, function='gaussian', k=500)
Step 5: Perform Geographically Weighted Regression (GWR)
gwr_model = mgwr.GWR(dependent_variable, independent_variables, w)
gwr_results = gwr_model.fit()
Step 6: Calculate predictions and residuals
predictions = gwr_results.predy
residuals = dependent_variable - predictions
Step 7: Calculate evaluation metrics
rmse = np.sqrt(mean_squared_error(dependent_variable, predictions))
mae = mean_absolute_error(dependent_variable, predictions)
r2 = r2_score(dependent_variable, predictions)
mape = np.mean(np.abs(residuals) / dependent_variable) * 100
Step 8: Print evaluation metrics
print("RMSE:", rmse)
print("MAE:", mae)
print("R-squared (R2):", r2)
print("MAPE:", mape)
Step 9: Display the predicted field with the data
gdf['Predictions'] = predictions
gdf.plot(column='Predictions', legend=True)
Step 10: Display the residual field
gdf['Residuals'] = residuals
gdf.plot(column='Residuals', legend=True)
according to Bard
import pandas as pd
import pysal as ps
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_squared_error, r2_score
Read data from the Excel file
data = pd.read_excel("C:\D\R1.xlsx")
Separate dependent and independent variables
y = data["Dependent Variable"]
X = data.drop("Dependent Variable", axis=1)
Create a spatial weights matrix
w = ps.weights.DistanceBand(data, 500)
Fit the GWR model
model = MGWR(y, X, w, kernel="gaussian")
Make predictions for the dependent variable
predictions = model.predict(data[["u", "v"]])
Calculate evaluation metrics
rmse = mean_squared_error(y, predictions)**0.5
mae = mean_absolute_error(y, predictions)
mape = mean_absolute_percentage_error(y, predictions) * 100
r2 = r2_score(y, predictions)
print("RMSE:", rmse)
print("MAE:", mae)
print("MAPE:", mape)
print("R2:", r2)
Create a new column with predicted values and one with residuals
data["Predicted Value"] = predictions
data["Residual"] = y - predictions
Display the predicted field with the data
print(data)
Display a field that represents the difference between the actual value and the predicted value (residual)
print(data["Residual"])

I just confirmed on my end that either of these will work:
from mgwr.gwr import GWR, MGWR
GWR
or
import pysal.model.mgwr as mgwr
 mgwr.gwr.GWR()

Closing #1287

where I need to stick this in my code below? could you please help me on this?
from mgwr.gwr import GWR, MGWR
GWR()
or
import pysal.model.mgwr as mgwr
mgwr.gwr.GWR()

import geopandas as gpd
import pysal.lib
import pysal.model
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from math import sqrt
from pysal.explore import esda
from pysal.viz import splot
Load the data from the Excel file
file_path = "C:\D\R1.xlsx"
data = pd.read_excel(file_path)
Prepare the data for GWR
y = data.iloc[:, 0].values  # Dependent variable
X = data.iloc[:, 1:-2].values  # Independent variables
coords = list(zip(data['u'], data['v']))  # Coordinates
bw = 500  # Bandwidth
kernel = 'gaussian'  # Kernel function
Fit the GWR model
model = pysal.model.GWR(coords, y, X, bw=bw, kernel=kernel)
results = model.fit()
Predict values using the GWR model
predicted_values = results.predy
Calculate RMSE, MAE, MAPE, and R-squared
rmse = sqrt(mean_squared_error(y, predicted_values))
mae = mean_absolute_error(y, predicted_values)
mape = np.mean(np.abs((y - predicted_values) / y)) * 100
r2 = r2_score(y, predicted_values)
print("RMSE:", rmse)
print("MAE:", mae)
print("MAPE:", mape)
print("R-squared:", r2)
Create a GeoDataFrame with the predicted values
gdf = gpd.GeoDataFrame(data, geometry=gpd.points_from_xy(data['u'], data['v']))
gdf['predicted'] = predicted_values
Calculate residuals
residuals = y - predicted_values
gdf['residual'] = residuals
Plot the predicted field and residuals
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
gdf.plot(column='predicted', cmap='viridis', ax=ax[0], legend=True)
ax[0].set_title('Predicted Field')
gdf.plot(column='residual', cmap='coolwarm', ax=ax[1], legend=True)
ax[1].set_title('Residuals')
plt.show()

@jamalnuman68108 please refer to the official documentation or other verified teaching resources (https://pysal.org/notebooks/model/mgwr/intro.html, https://martinfleischmann.net/sds/chapter_10/hands_on.html#geographically-weighted-regression) before asking developers to fix code generated by LLM.

The issue here is not who has created the code. This issue is that when it comes to spatial data all Apps get weak!
Even when you apply one single line of code, errors are there! Errors that never ends with the GWR
import pysal.model.mgwr as mgwr
from mgwr.gwr import GWR, MGWR

It seems to me that you are running some ancient version of pysal based on the trackback you show. Ensure you have PySAL and all the other packages in their latest versions.

This issue is that when it comes to spatial data all Apps get weak!

I would appreciate if you kept comments like this for yourself. PySAL has a Code of Conduct to follow.

It seems to me that you are running some ancient version of pysal based on the trackback you show. Ensure you have PySAL and all the other packages in their latest versions.

The pysal is update but the issue persists to exists as per the screenshots below

No, it is not. The latest version is v23.07. You have a version 2.0.0, which is from January 2019, nearly four years ago. Please refer to installation instructions..

No, it is not. The latest version is v23.07. You have a version 2.0.0, which is from January 2019, nearly four years ago. Please refer to installation instructions..

As per the screenshot below, I executed the command below with no luck to get the most updated version.
conda install --channel conda-forge pysal
there should be something else wrong

My suggestion would be to create a fresh environment using the following:
conda create -n pysal_env
conda activate pysal_env
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
conda install pysal

It is not wise to install stuff to the base conda environment and to use the defaults channel. We as developers do not have any control over versions available in defaults but we have full control over conda-forge.
I would need to debug your environment to understand what is happening and I don't really have time to do that.
Also, it is better to use conda list pysal to show the installed version than pip.

Despite the fact that the pysal is properly installed, I’m still getting more and more errors when it comes to working with GWR

The issue here is not who has created the code. This issue is that when it comes to spatial data all Apps get weak!

the issue is not who created the code per se, but rather that the original poster DID NOT create it. Regardless of whether the original code was written by an LLM, a friendly colleague, or lifted from stack exchange, the problem is that the OP does not fully understand what the code is doing, and thus has made some quick (and inaccurate) judgments about what could be going wrong. This is the trouble with trying to debug code you did not write yourself--it is hard to diagnose errors. As a consequence, the OP is asking our dev team to debug "errors" that arise because the LLM does not appear to be trained on the latest version of the PySAL source code.
In this case, it is very easy to see that the errors you are encountering, @jamalnuman68108, come from trying to use an old version of the software that is no longer supported (so that import structure will fail because GWR does not live in the spreg module) so the LLM has generated inadequate code which does not run. There is nothing wrong with the software but you have effectively asked "someone" (an LLM) to write a program that neither of you understand
In general, the pysal team does not have the resources to troubleshoot code written by an LLM because the LLM is likely to generate nonsense (as it did here). If you can't understand what the LLM is doing, then unfortunately we probably cannot help you

what is LLM here?
Regardless of whether the original code was written by an LLM

a large language model like chatgpt

Thank you for the input
Does this mean that the code below will never work anyway. I couldn’t figure out what is outdated, is it code or the software?
All what I need to do it do predict the value of first variable based on the second and third variables using the GWR. The XY coordinates are given.
Not sure why this should be very challenging while other machine learning models like XGBoost, ANN, RF, etc.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Gaussian kernel
def kernel(point, xmat, k):
m,n = np.shape(xmat)
weights = np.mat(np.eye((m)))
for j in range(m):
    diff = point - X[j]
    weights[j, j] = np.exp(diff * diff.T / (-2.0 * k**2))

return weights

def localWeight(point, xmat, ymat, k):
wt = kernel(point, xmat, k)
W = (X.T * (wt*X)).I * (X.T * wt * ymat.T)
return W
def localWeightRegression(xmat, ymat, k):
m,n = np.shape(xmat)
ypred = np.zeros(m)
for i in range(m):
    ypred[i] = xmat[i] * localWeight(xmat[i], xmat, ymat, k)

return ypred

data = pd.read_csv('tips.csv')
colA = np.array(data.total_bill)
colB = np.array(data.tip)
mcolA = np.mat(colA)
mcolB = np.mat(colB)
m = np.shape(mcolB)[1]
one = np.ones((1, m), dtype = int)
horizontal stacking
X = np.hstack((one.T, mcolA.T))
print(X.shape)
ypred = localWeightRegression(X, mcolB, 0.5)
xsort = X.copy()
xsort.sort(axis=0)
plt.scatter(colA, colB, color='blue')
plt.plot(xsort[:, 1], ypred[X[:, 1].argsort(0)], color='yellow', linewidth=5)
plt.xlabel('Total Bill')
plt.ylabel('Tip')
plt.show()
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Gaussian kernel
def kernel(point, xmat, k):
m,n = np.shape(xmat)
weights = np.mat(np.eye((m)))
for j in range(m):
    diff = point - X[j]
    weights[j, j] = np.exp(diff * diff.T / (-2.0 * k**2))

return weights

def localWeight(point, xmat, ymat, k):
wt = kernel(point, xmat, k)
W = (X.T * (wt*X)).I * (X.T * wt * ymat.T)
return W
def localWeightRegression(xmat, ymat, k):
m,n = np.shape(xmat)
ypred = np.zeros(m)
for i in range(m):
    ypred[i] = xmat[i] * localWeight(xmat[i], xmat, ymat, k)

return ypred

data = pd.read_csv('tips.csv')
colA = np.array(data.total_bill)
colB = np.array(data.tip)
mcolA = np.mat(colA)
mcolB = np.mat(colB)
m = np.shape(mcolB)[1]
one = np.ones((1, m), dtype = int)
horizontal stacking
X = np.hstack((one.T, mcolA.T))
print(X.shape)
ypred = localWeightRegression(X, mcolB, 0.5)
xsort = X.copy()
xsort.sort(axis=0)
plt.scatter(colA, colB, color='blue')
plt.plot(xsort[:, 1], ypred[X[:, 1].argsort(0)], color='yellow', linewidth=5)
plt.xlabel('Total Bill')
plt.ylabel('Tip')
plt.show()

In general, the pysal team does not have the resources to troubleshoot code written by an LLM because the LLM is likely to generate nonsense (as it did here). If you can't understand what the LLM is doing, then unfortunately we probably cannot help you


Serverside rendering

broken with webpack..

any details?

oh, its related to SSR not webpack 😉
see how they solved it here:
https://github.com/Olical/react-bodymovin

this issue is about ssr

This has become a bit of a confusing thread. Was SSR added already in the PR? If not where can we look to make the required changes?

There was an SSR issue with bodymovin. I believe this commit fixed it. https://github.com/decentfox/lottie-web/commit/85434e349d77a47fa65313fa9d06475ecb829464

Sorry, thought that was merged into master. This is the issue I faced. https://github.com/airbnb/lottie-web/issues/440#issuecomment-338057030

Any updates on this?

TY! Will have a look.


Outlook attachments should support complete name/bytes structure

Describe the Bug with repro steps

In GA designer, create a flow with the 'Send an email (V2)' action
Add the Attachments parameter and input random text into the Name and Content fields.
Save the flow
Open the same flow in the new designer
The attachments show as an invalid parameter ('Please enter a valid array')
If you delete that parameter and try to add attachments from the v3 designer, you can only add individual strings

Workflow JSON
No response
Screenshots or Videos
Old designer - can save this flow:

Open same flow in new designer. Flow loads but shows attachments are invalid:

If you delete that parameter and try to add attachments from the v3 designer, you can only add individual strings:

Browser
Tested in Chrome
Additional context
No response

@Eric-B-Wu

tracked-by:"Azure/LogicAppsUX#1996"

Closing since it's tracked by a more generic issue


Improving Support for the UP

Definition of work: understand where current support processes can be improved and update support materials for the UP
User value: the UP is a bigger, more complicated product than the covid pipeline with a lot of interest from users, there will be questions they have about it, we should be prepared to hand those support requests
Linked dependencies and appropriate labels: Engagement led, support team heavily involved
Outcome-oriented deliverable: updated support docs for UP issues, trained support team on commonly escalated issues

Support resolution research complete with top categories to improve identified here: https://app.box.com/s/bqhi4lcmqzy94vpca6tp0eivxpa9gp5c
Next step is training the support team on top issues identified

Further research that informed the final report here: https://docs.google.com/document/d/1u82Xu2oTb7L95GtGfDt1q-T2oGkdLtW_cxa6EAZoQS0/edit?usp=sharing


NMS.KK.DND.002 - Rules for deactivation related to DND (Do Not Disturb)

NMS shall mark a beneficiary as deactivated with associated reason if the beneficiary has its MSISDN number added to the DND database. This requirement applies to both cases listed below:

The beneficiary is called for first time
The beneficiary has added its MSISDN number to the DND database while the OBD service is ongoing

In case a beneficiary removes its number from DND database, then such a use-case is currently not handled by the software.

This comes into play when we get an entry in the CDR file indicating that the subscriber is in the DND -- in that case, we should set the beneficiary status (and subscription status?) to Deactivated.


🐛 [BUG] - Arrow on CTAs not animating on hover

Describe the bug
Expected: CTA arrow should animate on hover
Current: No animation on hover
Example page: https://resources.github.com/devops/tools/compare/
the unhovered arrow starts with the classes Primer_Brand__ExpandableArrow-module__ExpandableArrow___rkfek, and on hover has the classes Primer_Brand__ExpandableArrow-module__ExpandableArrow___rkfek Primer_Brand__ExpandableArrow-module__ExpandableArrow--expanded___gajDr, but looks like there's no corresponding class in the css bundle.
Reproduction steps
go to https://resources.github.com/devops/tools/compare, hover over the start a free trial CTA.

Expected behavior
Expected: CTA arrow should animate on hover

Screenshots
https://user-images.githubusercontent.com/20496719/195380869-d07bed09-5ba1-4429-93ee-4bcea0a959f1.mp4

Browsers
Chrome
OS
Mac

@ywenc - do you happen to have css tree-shaking enabled in your production build step?
https://user-images.githubusercontent.com/13340707/195406070-14d72ac2-954f-46ed-bfd8-fdb6405b134e.mov
I can't reproduce this in dev mode (see above).. but I'm also unable to build and inspect the prod bundles due to getting this error:
> Build error occurred
[Error: ENOENT: no such file or directory, rename '.../resources-hub/.next/export/webcasts/EMEA-Security-Learning-Journey-H2.html' -> '.../resources-hub/.next/server/pages/webcasts/EMEA-Security-Learning-Journey-H2.html'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'rename',
  path: '.../resources-hub/.next/export/webcasts/EMEA-Security-Learning-Journey-H2.html',
  dest: '.../resources-hub/.next/server/pages/webcasts/EMEA-Security-Learning-Journey-H2.html'
}

Am I missing something from Contentful?

@ywenc I pretty sure this issue is occurring due to our use of Purge CSS. I wonder if we need to add exceptions or do something like pipe in PrimerBrand's compiled CSS into /script/utils/purgecss.js?

@ywenc I'm going to close because I believe we need to fix this in Resources Hub


Add 'forgot password' feature

Description
Should be possible for users to reset PW when they forgot theirs.

Require https://github.com/codescalers/CSHR/issues/351


Update Blog “2021-02-28-easy-excerpt-from-content”

Automatically generated by Netlify CMS

👷 Deploy Preview for cerulean-cat-88c204 processing.

Name
Link

🔨 Latest commit
5419f735d66e3c54feb6fd1174657644df4df5a8

🔍 Latest deploy log
https://app.netlify.com/sites/cerulean-cat-88c204/deploys/62a580fe8ab5b1000994a851


Ensure that version in the Makefile are updated so tests pass

Do we have versions of calico/test too?  Or should that always be latest?
If the latter, then LGTM.


ThreadPool::install and thread overhead

I've made a reproduction of an issue with ThreadPool::install and the overhead I see due to the install call itself. In the reduced problem, the function that runs in ThreadPool::install() is entirely serial, and the overhead is proportional to the number of threads in the pool.
Maybe this is totally overblown, the overhead is on the order of milliseconds, but I had hoped to be able to parallelize problems are that fast to compute
Example
Code on the branch rayon-pool-install-201812 in ndarray.
You can clone it and run the test like this:
NUM_THREADS=1 cargo run --release --example threadpool_install  --features=rayon
NUM_THREADS=16 cargo run --release --example threadpool_install  --features=rayon

The program is in examples/threadpool_install.rs and it calls ThreadPool::install repeatedly. The work inside the install takes about 3 milliseconds on its own. (Does this sound short?)
I ran perf on the example executable with 16 threads and it says that rayon functions use significant time:
  61,53%  threadpool_inst  threadpool_install  [.] ndarray::impl_methods::<impl ndarray::ArrayBase<S, D>>::mapv_inplace
  13,17%  threadpool_inst  threadpool_install  [.] <std::thread::local::LocalKey<T>>::with
   9,35%  threadpool_inst  threadpool_install  [.] rayon_core::registry::WorkerThread::steal
   7,56%  threadpool_inst  threadpool_install  [.] crossbeam_epoch::default::HANDLE::__getit
   5,92%  threadpool_inst  threadpool_install  [.] <crossbeam_deque::Stealer<T>>::steal
   0,94%  threadpool_inst  libc-2.24.so        [.] _int_free
   0,37%  threadpool_inst  threadpool_install  [.] crossbeam_epoch::guard::Guard::new
   0,36%  threadpool_inst  [kernel.kallsyms]   [k] entry_SYSCALL_64
   0,32%  threadpool_inst  threadpool_install  [.] crossbeam_epoch::internal::Global::try_advance
   0,28%  threadpool_inst  threadpool_install  [.] rayon_core::registry::WorkerThread::wait_until_cold

If I change the thread pool to use 1 thread, I get a serene result, 99% of execution time is in the actual job:
  99,95%  threadpool_inst  threadpool_install  [.] ndarray::impl_methods::<impl ndarray::ArrayBase<S, D>>::mapv_inplace
   0,03%  threadpool_inst  [kernel.kallsyms]   [k] entry_SYSCALL_64
   0,01%  threadpool_inst  ld-2.24.so          [.] do_lookup_x
   0,00%  threadpool_inst  libc-2.24.so        [.] _dl_addr
   0,00%  threadpool_inst  libpthread-2.24.so  [.] __pthread_mutex_unlock_usercnt
   0,00%  threadpool_inst  libc-2.24.so        [.] _int_malloc
   0,00%  threadpool_inst  libpthread-2.24.so  [.] pthread_cond_destroy@@GLIBC_2.3.2

Actual Use Case
Of course, if I don't have anything parallel to do, I can avoid using rayon. I saw this problem when using a too big thread pool with just a few parallel jobs. My use case is so far just an experiment, and I was doing this:

Using a ThreadPool with 4 threads; the ThreadPool is owned by a lazy static for the duration of the program.
Using Pool.install(|| f() ) around all my loops
Running a parallel iterator of length 2

The parallel iterator was only length 2 due to the problem's size. Here I noticed that a ThreadPool of 2 threads would have less overhead. With 4 threads for 2 jobs, the overhead would eat up any gains from parallelism.
The program should use 1 thread per physical core and can't adapt the number of threads (creating the thread pool is too slow). It needs to be ready for tasks that can be split into 2 or more parallel jobs.
I also seem to be able to reproduce this issue with the global pool. Just rayon::join with two parallel jobs in the whole program, and if I run this with 2 threads, I get a speedup from 95µs to 61µs, but the speedup disappears if there are more threads in the global pool. This makes it seem that rayon can parallelize this problem well!

This issue seems related. #576

Yeah, we need to improve the situation with underutilized thread pools.

I think you could emulate this with crossbeam::scope or scoped_threadpool, but not quite as neatly packaged as rayon::join.  I'm sure they could add such a convenience function though.
I'm not sure what you want rayon to do without stealing -- just run serially?  Forcibly send the second job to another thread in the pool?  The latter would require targeted wake-ups, which basically gets back to the main issue of #576.

Neither of them seem to support jobs stored on the stack, though.
I'm writing some experiments that take me closer to realizing what kinds of problems are involved, right now breaking out rayon's join into a PoC where it directly sends stack jobs to other threads in a pool. Not that Rayon should solve it that way, but it's a learning experience opening to work on. I don't know Rayon, but it sounds like #576 is pretty much the same issue.

Neither of them seem to support jobs stored on the stack, though.

Oh, I thought you just wanted scoped lifetimes. I guess you mean like our StackJob that doesn't touch the heap at all?  In crossbeam's case, a little heap allocation will be dwarfed by dynamically starting a thread, but maybe scoped_threadpool could reasonably implement a more direct join.

I noticed the following lines in rayon's scheduler:
let num_to_wake = std::cmp::min(num_jobs, num_sleepers);
self.wake_any_threads(num_to_wake);

It seems if we create 5 jobs, then 5 threads get woken at once. That may be a bit excessive if those jobs are small. It's possible that just a single thread will quickly process all 5 jobs and then the remaining 4 threads have nothing to do. In other words, more time is spent waking threads than actually processing jobs.
A strategy some other schedulers (e.g. the Go scheduler) employ is to wake up just 1 thread if there are scheduled jobs. Then, if that thread finds a job for itself, it wakes another thread. Then that thread does the same thing, and the procedure goes on like that. So if we have 5 small jobs that are enough for a single thread, we'll only wake up one additional thread in vain.

@stjepang I think you have a good point -- however, in practice I believe this only ever sees one job at a time. While Registry::inject can take an arbitrary slice of jobs, all of its current callers only pass a singleton slice. The other waker is WorkerThread::push, which is only dealing with one job as well.

Then, if that thread finds a job for itself, it wakes another thread.

We do have this heuristic already in Sleep::work_found:
https://github.com/rayon-rs/rayon/blob/8c9ada2e0ad194692ccf9b75f4e36f356591ed4f/rayon-core/src/sleep/mod.rs#L95-L98
And here's AtomicCounters::sub_inactive_thread:
https://github.com/rayon-rs/rayon/blob/8c9ada2e0ad194692ccf9b75f4e36f356591ed4f/rayon-core/src/sleep/counters.rs#L160-L163
But they seem to be conflicted between waking "another thread" (1) versus "any sleeping threads" (capped at 2).

#746 has been merged! That means it is time to revisit this -- I haven't done so yet.

@cuviper it looks we are starting 2 threads. I know I was tinkering with that as a way to help us in "catching up" to spikes of work faster. I can't remember how effective these changes were, we should measure, but my memory is "a little but not much". I presently feel it's probably not worth it, and better to just wake one thread.

I've come back to testing newer rayon. Not with the exact same testcase yet (I'm sorry).
Some aspect of the same problem is still visible. Same has has been formulated in newer issues: high cpu usage when there are more threads than tasks. Let's say for example 8 threads in the thread pool but only work enough for four of them. With enough threads, the extra threads eat up the whole real time that could have been gained on the task, unfortunately.

The current conclusion of my experiments is a published crate for a new thread pool (there are too many crates for this already!) - called thread-tree which is a very simple binary tree of worker threads. It seems to work like I want for the benchmarks I run.
The problem that I saw both with rayon and to some extent with my own thread pool experiments was that there is overhead in the situation when there are two work items to run, but four threads contend to pick it up.
If I have a binary tree of worker threads, then I always have a 1-1 channel sender/receiver to each thread, so this effect of contention between workers doesn't happen. Maybe it's an unfair conclusion, because in the MVP version of the implementation I only support 1, 2 or 4 parallel jobs.
I have a question for @nikomatsakis  and @cuviper: This new crate is partly based on code that I took from rayon-core. That was very useful for me, wouldn't have been possible otherwise (StackJob and its execution) - is it enough that I mention you as authors in a notice in the code and in the Readme. Is it ok if I also mention you in the Cargo.toml authors list? Or maybe that's not neeed?
Thread-tree crate: https://docs.rs/thread-tree/
Matrixmultiply explanation: https://github.com/bluss/matrixmultiply/pull/52

The case described in the original case is fixed by the new scheduler, so I'll close. I've been wanting to close this anyway. Even if it looks like I'm going for another solution due to similar issues.

Thanks for the followup @bluss!

Is it ok if I also mention you in the Cargo.toml authors list? Or maybe that's not neeed?

If it were a close fork then shared authorship might be nice, but I think it's not needed since you just took a small piece, and the crate is substantially different.
I wish Rayon could be everything to everyone for parallelism, but I think it's OK to accept that different designs will be better for some workloads.


SBT runs everything twice in watch mode

problem
From within SBT, when I run anything in watch mode (often ~testOnly xxx but applies to anything really), SBT executes the command twice per change over 90% of the time.
expectation
Changing one file should result in one execution, not two.
environment

SBT 1.3.10 although this has been happening for a long time now
(Arch)Linux 5.6.13
FS is ext4
Nearly always making changes via IntelliJ

steps
Unfortunately I don't know how I can provide a reproduction for this. Please advise? Is there some kind of properly that I can enable in SBT to provide diagnostic information?

There are two things you can do to help debug. One is to set watchLogLevel := Level.Debug. The other, more verbose, thing is to just type debug in the sbt shell. You should see detailed information about what triggered the build. With the debug logging, you can tell what kind of file event triggered the build (i.e. Creation, Deletion or Update) which can be helpful to figure out what is happening.
Watch should only rebuild if the file hash has changed since the last time it has triggered. One thing that can cause problems is how the editor implements save. A lot of editors will implement save with delete + rename. This of course causes the file hash to change. sbt has a parameter watchDeletionQuarantinePeriod that will prevent deletions from immediately triggering and waits 50 milliseconds by default to see if another write to the file follows the deletion. So one thing to try would be to increase that parameter.
But this is not the only way that editors implement save. Apparently vscode does something weird with some that bloop works around in their watch implementation: https://github.com/scalacenter/bloop/blob/118a551b935f82ebe6a785818967bf8a3ca27597/frontend/src/main/scala/bloop/io/SourceWatcher.scala#L75
We may want to do something similar in sbt but I have no idea if that would help you at all.
FWIW, I have used sbt 1.3.x extensively with intellij on mac os and have not had issues. So I don't know if that's a platform thing, a filesystem thing or possibly just a disk/cpu speed thing.

Thanks a lot for that info @eatkins ! I'll try those settings, collect and inspect some logging and get back to you. Cheers!

Hey @eatkins. I've captured the debug logging of a case where it invokes twice when only one file is changed: https://gist.github.com/japgolly/3f5df793d45e5caf4d7649f79b5995ad

I inserted a little separator at line 1189 to highlight the gap between invocations
the file I changed is MockInterpreters.scala
the first pass correctly detected that MockInterpreters.scala is the change
the second pass is apparently caused by External API changes: API Changes: Set(NamesChange(...
There a lot of noise like The shipreq.base.util.Optics$ has the following regular definitions changed: UsedName(finalize,[Default]), UsedName(lensTuple2_1,[Default]) doesn't make sense to me. Optics.lensTuple2_1 is a method I added yesterday and haven't touched since. It seems like a bug that it's appearing in today's logs as a change.

I can also confirm that ThisBuild / watchDeletionQuarantinePeriod := 500.milliseconds doesn't avoid the issues that I've having, unfortunately.

I completely forgot that there is another parameter, watchAntiEntropy, that configures the minimum amount of time that must elapse after a particular file has triggered a build before that same file can trigger a new build. Based on the logs, I suspect increasing that parameter would fix your problem. You can see in the logs that the service detected three updates to the file after the trigger and it discarded the first one because it occurred 21ms after the trigger. The default value is 500ms so maybe try bumping that? Unfortunately the watch logs don’t currently show the modification time stamps so I can’t tell how long the gap was. It does appear though that the event that triggered the rebuild appeared seven seconds or so before the build itself so I have some reason to think the anti-entropy parameter could resolve this issue. I’d also be curious as to why the file is being modified four times after you save it.

What are the versions of IntelliJ and Scala Plugin, and do they try to format your code on save?

Ok I've found the issue! I have a few modules with config like this:
  lazy val webappBaseTest =
    crossProject("webapp-base-test")
      .jsSettings(jsDependencies in Test += ProvidedJS / "webapp-base-test.js")

And a shared (static) JS file shared using symlinks. Example:
> la webapp-base-test/js/src/test/resources/
total 8
drwxr-xr-x 2 golly users 4096 May 20 18:35 .
drwxr-xr-x 4 golly users 4096 May 27 19:00 ..
lrwxrwxrwx 1 golly users   54 May 20 18:35 webapp-base-test.js -> ../../../../../frontend/dist/local/webapp-base-test.js

Now the symlink target isn't changing, nor are the symlinks changing or being recreated. However if I look at the debug SBT logs I'm seeing double compilations like this:
~tc
[info] Compiling ...
[error] Total time: 6 s, completed 31 May 2020, 4:13:46 pm
[info] 1. Monitoring source files for webapp/tc...
[info]    Press <enter> to interrupt or '?' for more options.
[info] Build triggered by .../webapp-base-test/js/src/test/resources/webapp-base-test.js. Running 'tc'.
[info] [watch] [debug] Received file event Trigger for Creation(.../webapp-base-test/js/src/test/resources/webapp-base-test.js, 2020-May-31 16:13:40.811 AET).

[info] Compiling...

It seems the watch code is treating the symlink as a creation every time it encounters it.
(And thanks @eed3si9n but I'm not using any auto-formatting)

@japgolly I definitely buy that symlinks are causing the problems. Would it be possible to produce a minimal, but complete repository, that reproduces the issue?

Yes! Let me get back to you.

* There a lot of noise like `The shipreq.base.util.Optics$ has the following regular definitions changed: UsedName(finalize,[Default]), UsedName(lensTuple2_1,[Default])` doesn't make sense to me. `Optics.lensTuple2_1` is a method I added yesterday and haven't touched since. It seems like a bug that it's appearing in today's logs as a change.

I found this ticked when was googling for this has the following regular definitions changed. This is also happening in my work project. Any hint about this @eatkins or @eed3si9n ?

@jtjeferreira the quoted log message does not originate in the watch subsystem. It comes from https://github.com/sbt/zinc so I'd recommend filing an issue in that repo.


Double Wrap Locked Builds

This PR fixes #99 by replicating the layers of Monster() in the normal build:

When test is created
When test become Best

This double-wrapping of the original mon seems to be the root cause of #99 so this double-wraps a locked build to achieve the same internal state.

Unfortunately, I appear to have re-introduced a different issue where the original runes aren't getting locked.

OK I think I figured it out.  Locked loadouts are made from the runes in Mon.Current.Runes.  When we reload, we orphan all of those runes.  This puts the system in a state where the right runes aren't getting unlocked/reused/locked correctly.  Testing a fix.

For general architectural consistency, I went ahead and returned the runes from Save in GetRunes and used ApplyRune in GenBuilds to link them to a loadout in the same way the system appears to.  The extra effort was probably unnecessary but it should minimize future bugs from refactors.  I also rebased for convenience.


Enigma Rework

Gravity Well: Removed entirely. Singularity moved to Black Hole.
Malefice:

Now always pulls target enemy towards the strongest source: Black Hole, then Midnight Pulse, then Enigma.
Each 25% health that the enemy has when applying Malefice on it increases the duration by 2 seconds.
Eidolons' attacks causes the next instance of Malefice's stun to be increased a bit.

Midnight pulse:

Now always pulls all targets in range towards it, if they're not affected by Black Hole.
Eidelons in Midnight Pulse gain bonus health regeneration. This effect lingers for 3 seconds after leaving Midnight Pulse's AoE.

Black Hole:

Black Hole constantly pulls all enemies in a 1200 AoE towards it. The closer they are to the Black Hole, the stronger the pull. The pull has a minimum of 75 pull power.
Each enemy hero that it caught within Black Hole increases Singularity stacks by 1 for each damage instance. Singularity stacks are consumed on the next Black Hole cast, increasing the pull power by 20 per stack, the Black Hole's AoE by 25, and the pull AoE by 50.

Demonic Conversion:

Shrink tooltips:
Transforms a creep into 3 eidolons.  Eidolons heal completely and multiply after 6 attacks.  ENDLESS: Eidolons can multiply infinitely, but consecutive splits require 3 more attacks.  PRIMORDIAL SHARD:  Eidolons gain a percentage of Engima's Health, Armor, Damage, Attackspeed, and Movespeed.  VOID MASS: Eidolons' attacks amplify the pull effects of Engima's spells by 7%, this stacks between Eidolons, but indivudal Eidolons' attacks only re-apply the effect.

TALENTS:
Remove:
+2 malefice duration
+225 black hole cast range
+15% eidolons stats
+80 Black Hole radius
+15% Gravity Well
+1 Black Hole duration
Change:
Malefice is applies on all enemies in 215 range -> Malefice is applies on all enemies in 400 range.
Apply:

Each Singularity stack consumed now decreases Black Hole's cooldown by 2 seconds.
When Enigma dies, a Blackhole is cast on its location of death with half duration. Does not consumes Singularity stacks.
Midnight Pulse doubles its AoE over the duration.
Enigma has a constant AoE Midnight pulse around him. Stacks with regular pulse.
True malefice - malefice now triggers Black Holes instead of just stunning.
Whenever Enigma casts a spell, his void energy grants him magic immunity for one second. Duration stacks with itself.

Some questions

Does this work retroactive?
Does this collect Singularity stacks?
What does this mean?

Done!


Campaigns API -- some Action Guides are returning null properties

BUG
Current Behavior

Not all properties are returned from API for some Action Guides

I'm seeing null values for some Action Guides -- Birthday Mail's calling script is one example, there's content that appears on web, but the properties are NULL in the API. Other Action Guide nodes are returned as expected for Birthday Mail.

I also the see same behavior on staging when after creating a new Action Guide and adding it to the Yeah Science campaign: https://staging.dosomething.org/api/v1/campaigns/1377

The Yeah Science Action Guide node displays as both English, Global translations available:

Steps to Replicate

Happens every time on Birthday Mail on prod so far, haven't found others yet

Why This Matters

Action Guides won't properly render in mobile app

Hmmmm right before my very 👀 the Birthday Mail calling script response has changed to return content

I tried clearing cache on staging to see if it fixes Yeah Science, but its properties are still set to null.

LOL. Yeah I just loaded it... in the first screenshot, it actually looks like the copy: {...} property was just collapsed is all...

If it still happens I suspect maybe a caching thing, but I do remember making it so that if a campaign is edited, it clears that campaign from the cache in the API... but it was also a while back and my memory is fuzzy 👴


Error:Hub Controller process exited prematurely with signal 'SIGABRT'.

Storage Explorer Version: 1.15.1
Build Number: 20200902.9
Platform/OS: MacOS Catalina
Architecture: x64
Bug Description
The requested action could not be completed.
Details:
["Error:Hub Controller process exited prematurely with signal 'SIGABRT'."]

Steps to Reproduce

Just open it on MacOS.

Expected Experience
To actually work
Actual Experience
Doesn't work

@JCKodel I did upgrade the version of storage explorer to Version 1.15.1 and it is working fine.

@JcKodel can you:

In Storage Explorer settings, turn on Verbose Authentication Logging under, Application -> Sign-in
Close Storage Explorer
Delete ~/.ServiceHub/logs
Start Storage Explorer
Do everything you do to reproduce this problem
After it reproduces, zip up ~/.ServiceHub/logs, and attach it here

Thanks.

@MRayermannMSFT
Retrying the operation generates no log (I guess I need to restart Storage Explorer for the log option to activate?), but doing so fixes the problem (until the next day or so).
If I click on Reenter credentials, the "Connect to Azure Storage" just freezes (the circular progress keep spinning, but nothing happens).

After quitting and reopening, the following error immediately appears:
Unable to retrieve child resources.

Details:
["Error:Unable to retrieve authentication token."]

At this point, this is the log:
1.zip
Now I deleted all log and clicked Reenter credentials (now the amazing Microsoft Login takes places... such a simple task, so poorly implemented):
First, I pick my account (I have this account since 2004  or 2005, I guess, it's ancient).

Then, for no reason, I'm redirected to GitHub ¬¬

Then, I need to reenter my e-mail and 20 characters password, EVERY FREAKING TIME.

Finally, because of the will of Odin, I'm redirect to login in a company that I don't work for about 1 year now.

I know clicking my e-mail won't do nothing, but, what other option do I have? Doing so, it redirects me to GitHub (WTF?????)

And I have to retype my e-mail and 20 character password (I'll just copy and paste the screenshot above, it's the same)

2FA kicks in again and I have to, one more time, authorize it from my mobile phone.
All that to get to this:

Just tell me, HOW IN THE HELL do you guys can mess up a simple login??? Get my e-mail, password and 2FA and authenticate me! It's not that hard!
Now the only option I have is to cancel this operation, closing the authentication window. This is the log so far:
eebbbc5e-AadAccountProviderClient-4657-8pZG7I-4657-1.log
(there is only one file)
And now, the grand finalle! After quitting Storage Explorer and reopening, ta-da! It's authenticated! So easy!
After this walk in the park, very easy and uneventful authentication, this is the log:
Archive.zip

Glad it eventually worked. Sorry you had to go through such a frustrating experience. We'll keep working on this scenario.


CNET videos - extractor error

I am currently trying to download a video from http://www.cnet.com/videos/whiny-pothole-tweets-at-local-government-when-hit-by-cars-tomorrow-daily-187/ but I am getting an extractor error. Thanks for the help
[debug] Command-line args: [u'--write-thumbnail', u'-v', u'-f[width <=? 720]', u'-o', u'/media/56527b93-d25d-44e3-b738-f989ce2e49ba.%(ext)s', u'http://www.cnet.com/videos/whiny-pothole-tweets-at-local-government-when-hit-by-cars-tomorrow-daily-187/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.05.20
[debug] Python version 2.7.9 - Linux-3.19.0-18-generic-x86_64-with-Ubuntu-15.04-vivid
[debug] exe versions: ffmpeg 2.6.git, ffprobe 2.6.git, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: An extractor error has occured. (caused by KeyError(u'rtmp',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 273, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/cnet.py", line 45, in _real_extract
    vid = vdata['files']['rtmp']
KeyError: u'rtmp'
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 649, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 279, in extract
    raise ExtractorError('An extractor error has occured.', cause=e)
ExtractorError: An extractor error has occured. (caused by KeyError(u'rtmp',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output

Thanks for the report. In the next version this problem should have been fixed.


Separate Auto Lossy and Lossless

Right now a source png image will only be compressed in auto output mode if the browser supports webp, otherwise it will stay as a png.  It would be useful either to have separate auto lossy and auto lossless output modes to change the fallback order to webp->jpg->png, or to have the auto output mode choose a fallback order depending on the presence and value of the quality setting (that is, if a user asks for q_85 assume that they want auto lossy mode).

Hello @TiKevin83 ! that's a quite useful use case.
I'll add it to 1.2 milestone
thanks for the feedback


Add cluster disruption tests

Summary of the changes / Why this improves CrateDB
Checklist

[ ] User relevant changes are recorded in CHANGES.txt
[ ] Touched code is covered by tests
[ ] Documentation has been updated if necessary
[ ] CLA is signed
[ ] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Tests with @Repeat are passing https://jenkins.crate.io/job/CrateDB/job/crate_on_pr/4996/

retest this please

@mfussenegger pushed the slow tests annotation. Can you please have another look?


Is okay to change AGENT_NAME_MAX_LEN value as we need?

Possible to increase it to 40 locally? And build .jar files for our own environment.
Thank you,

Hi tinabear,
I think it's ok. But You should build agent and collector together. Maybe web ...

Hi tinabear,
I would like to add, if you modify AGENT_NAME_MAX_LEN, old data in hbase will be invalid. Please see the two classes: TraceRowKeyDecoderV1/V2.  Pinpoint can not decode rowkey that AGENT_NAME_MAX_LEN changed.


Remove IDs from OrderParams for ServiceOffering/ServicePlan

[ ] depends on https://github.com/ManageIQ/topological_inventory-api/pull/243

generated, description of problem in api PR.

@agrare please :eyes:

cc @syncrou


Attempting improvements on DirectRunner Python dash

Updating charts for Python DirectRunner benchmarks.
Sample image - note how both series are shown in a single chart.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

r: @tysonjh


Contribution guide part I

This PR starts work on #355 by adding an incomplete contribution guide to a new feature branch contribution-guide. To avoid monolithic pull requests, I wanted to open up the current work in progress for review, with an eye towards a second PR onto live once the rest of the content has been finished.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 62a90c7:
:clock10: Pending: waiting for processors (6 builds ahead of you)
:warning: Docs Build is busy, currently there are 6 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit 62a90c7:
:warning: Validation status: warnings

File
Status
Preview URL
Details

articles/contributing/pull-requests.md
:warning:Warning

Details

articles/contributing/code.md
:white_check_mark:Succeeded

articles/contributing/docs.md
:white_check_mark:Succeeded

articles/contributing/index.md
:white_check_mark:Succeeded

articles/contributing/reporting-bugs.md
:white_check_mark:Succeeded

articles/contributing/toc.yml
:white_check_mark:Succeeded

articles/media/git-workflow-step0.png
:white_check_mark:Succeeded

articles/media/git-workflow-step1.png
:white_check_mark:Succeeded

articles/media/git-workflow-step2.png
:white_check_mark:Succeeded

articles/media/git-workflow.vsdx
:white_check_mark:Succeeded

articles/media/pull-request-header.png
:white_check_mark:Succeeded

articles/contributing/pull-requests.md

Line 42: [Warning] Invalid file link:(~/media/git-workflow-step0.png).
Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit b587140:
:warning: Validation status: warnings

File
Status
Preview URL
Details

articles/contributing/pull-requests.md
:warning:Warning

Details

articles/contributing/code.md
:white_check_mark:Succeeded

articles/contributing/docs.md
:white_check_mark:Succeeded

articles/contributing/index.md
:white_check_mark:Succeeded

articles/contributing/reporting-bugs.md
:white_check_mark:Succeeded

articles/contributing/toc.yml
:white_check_mark:Succeeded

articles/media/git-workflow-step0.png
:white_check_mark:Succeeded

articles/media/git-workflow-step1.png
:white_check_mark:Succeeded

articles/media/git-workflow-step2.png
:white_check_mark:Succeeded

articles/media/git-workflow.vsdx
:white_check_mark:Succeeded

articles/media/pull-request-header.png
:white_check_mark:Succeeded

articles/contributing/pull-requests.md

Line 42: [Warning] Invalid file link:(~/media/git-workflow-step0.png).
Line 46: [Warning] Invalid file link:(~/media/git-workflow-step1.png).
Line 50: [Warning] Invalid file link:(~/media/git-workflow-step2.png).
Line 54: [Warning] Invalid file link:(~/media/pull-request-header.png).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Pioneer: Change event name to correspond a new one

Fixes #300

Download:
Environment: QT=5.12, ARCH=32, PYTHON_PATH=C:\Python35
installer/trik-studio-installer-master-i686-auto.zip
installer/trik-studio-installer-master-i686.exe
trik-studio-master_qt5.12_i686.zip
Environment: QT=5.12, ARCH=64, PYTHON_PATH=C:\Python35-x64
installer/trik-studio-installer-master-x86_64-auto.zip
installer/trik-studio-installer-master-x86_64.exe
trik-studio-master_qt5.12_x86_64.zip


Catalog: Allow "Starring" of items

Should be stored permanently:

Ideally, this means we want to add a userSettings backend service, which is cool, but maybe not within Phase 2?
As a middle ground we can for now store this information in the local storage, which will result into "starred" items being starred only for current browser for user

@shmidt-i I think we should lift over our internal settings API which can use localStorage as a persistent store.

Pausing work on this until next sprint.


Temoa: Expands transportation sector, specific nuclear turbine

This PR adds two iterations on the temoa UIUC model:

Diesel and E85
Specific turbine for the nuclear reactor

Abbott is a combined heat and power plant which has a much higher thermal efficiency than a nuclear reactor.

Closes #43

Hitting the merge button


Add Certificate refs for https listeners

Signed-off-by: EdgeJ<EMAIL_ADDRESS>
Description of your changes
This change adds references and selectors for ACM certificates to automate the creation of HTTPS listeners.

I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

Created and deleted listeners using example YAML.

Looks like e2e tests are failing for unrelated reasons, due to problems cleaning up:

Not sure why e2e tests are failing here. Setup problems?

@EdgeJ they are failing due to crossplane/crossplane#2944 -- fix is incoming 👍🏻

Rebased on master and pushed. Seems like it's fixed.

@EdgeJ they are failing due to crossplane/crossplane#2944 -- fix is incoming 👍🏻

Rebased on master and pushed. Seems like it's fixed.


Sync eng/common directory with azure-sdk-tools for PR 2972

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/2972  See eng/common workflow

/check-enforcer override


Error: Models based on 'Qwen2ForCausalLM' are not yet supported

What is the issue?
When I ran a this command:- ollama create qwen-14b-quant -f /home/vidyasagar/rr/ModelFile.txt
the text file consists of:-
FROM /home/user/rr/quantized_14_b_awq
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """"""
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
This is the terminal output:-
transferring model data
unpacking model metadata
Error: Models based on 'Qwen2ForCausalLM' are not yet supported
But I see there are the Qwen 1.5 supported models, Then why mine is not working?
can someone explain this? am I doing anything wrong?
OS
Linux
GPU
Nvidia
CPU
AMD
Ollama version
0.1.32

Afaik the Qwen 1.5 versions on Ollama are Qwen 2, look at family in the model layer and it says qwen2, might Ollama doesn't support the AWQ versions. 🤷♂ I tried converting the 7B AWQ version with llama.cpp and that didn't work either and it doesn't seem to be an official GGUF of the 14b AWQ version, which model did you use?


Optimize the concat_ws function

Signed-off-by: remzi<EMAIL_ADDRESS>Which issue does this PR close?

Closes #3856.
Closes #3857.
Rationale for this change

Simplify the concat_ws expression:

fold the whole expression to null if the delimiter is null
filter out null arguments
use concat to replace concat_ws if the delimiter is an empty string
concatenate contiguous literals if the delimiter is a literal.

What changes are included in this PR?

Are there any user-facing changes?

Thanks @HaoYang670


add interactions between partial and overedge

add interactions between partial and overedge to functions doco, overall doco?

Add the partial and overedge option detail in a control flow chart here: https://github.com/GeoscienceAustralia/dea-conflux/blob/main/doc/dea-conflux-control-flow.svg


feat: automatically convert image to tensors in TF data pipeline

See #1314.

@wjones127 added a test for all three image types


Unable to read azure keyvault secrets as environment  variable into ARO

We are trying to access azure keyvault secrets with ARO pod as environment variables. We have installed respective drivers as well.
our secret provider class:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
   name: azure-test
spec:
   provider: azure
   secretObjects:
     - data:
         - key: azure-test-read
           objectName: test-123
       secretName: azure-test-read
       type: Opaque
   parameters:
     usePodIdentity: "false"
     useVMManagedIdentity: "false"
     userAssignedIdentityID: ""
     keyvaultName: "AZP-test-KV"
     objects: |
       array:
         - |
           objectName: test-123
           objectType: secret
           objectVersion: ""
     tenantId: xxxxxx-xxx-4xxxx-xxx-xxxx

$ oc apply -f SecretProviderClass.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/azure-test unchanged
we have deployed the pod as well:
pod:
kind: Pod
apiVersion: v1
metadata:
   name: kv-access-sample-pod-3
spec:
   serviceAccountName: test-azurekeyvault
   containers:
   - name: ubuntu
     image:-ubuntu:20.04
     command: ['sleep', 'infinity']
     env:
       - name: test
         valueFrom:
           secretKeyRef:
             key: azure-test-read
             name: azure-test-read

while checking the event of the openshift UI:
PodPkv-access-sample-pod-3
NamespaceNSocpxxxxx
Just now
Generated from kubelet on nebula-f7jvg-worker-westeurope2-zxlrm
4 times in the last 0 minutes
Error: secret "azure-test-read" not found
I am getting this error.
like its unable to create the secret  in the ARO secret manager while deploying the secret provider class.
Cluster type: ARO

@san6789 To sync as Kubernetes secret, the volume mount is currently required. Please check the note in the doc.
We're working on a proposal to decouple the features, but for now you'll need to configure the volume, volume mount with the CSI driver to be able to make use of the sync secret capability.


Async compute work submitted at different time between replayer & native app

Sometimes there is a mismatch as to when async compute submissions happen. The app submits it at time X, but the replayer submits it at time Y. We need to approximate them to be closer for accurate perf analysis.

Do you have some more information on when or how this happens?

Caveat, I am more familiar with Vulkan capture and this issue may refer to D3D12, but because we synchronise threads implicitly inside the fwrite, even with a single core, if there are multiple threads recording, there is a chance of preemption between a call being forwarded down the chain to the driver and being recorded into the trace. During that preemption, a second thread could issue an API call down the chain and have it recorded in the trace. When the original thread is next scheduled it can issue the fwrite to record its call with the result that the trace contains them in a different order to the original one that they reached the driver.
If the new option implemented here (https://github.com/LunarG/gfxreconstruct/pull/1049) makes the ordering difference go away, the source of it may have been the mechanism I describe above. That's very much a hammer to crack a nut kind of workaround though and may radically change profiling results in its own way.
Multi-threaded replay should be a more robust solution, since the reordering we currently are liable to is inter-thread, but we do preserve ordering intra-thread. We should probably crack on with this approach since I can imagine situations with e.g. a resource upload thread and a render thread reordering synchronisation intermittently that lead to flat out bugs.
Caveat 2: someone should probably check my thinking here. What I wrote is based on eyeballing the code, not on experimenting and proving that the situation described occurs.


config option: allow AUTHENTICATE to bypass PASS

Both pidgin and hexchat appear to natively support only a single password field. This means that it's hard or impossible to connect to a server that requires a server password (with PASS) and then subsequently authenticate with SASL. (With hexchat, there's a workaround where you can hardcode the SASL PLAIN sequence as "custom commands" to be run after connecting.)
Perhaps it would make sense to add a config option that allows a client that successfully authenticated with AUTHENTICATE to skip the PASS command? Then these deficient clients could be configured to authenticate with SASL only.

This plan will not work as described because those clients send USER and NICK before AUTHENTICATE, and if they send those commands while unauthorized, we'll disconnect them.
I'm considering a workaround where they're allowed to send USER and NICK, but they're still considered unregistered until they send either PASS or AUTHENTICATE within 60 seconds, and if they don't they time out and are disconnected as usual. I think the only adverse implication of this is that clients without the password can "squat" nicks, denying others access to them (even though they actually use them)?

<dan> hmm
<dan> the password should only be checked when the user registration finishes
<dan> since CAP pauses registration, that works to our advantage
<dan> what we can do is have a flag where instead of just checking for a successful PASS when reg has completed, check for "successful PASS or SASL login"
<dan> if either of those are done, then let them in
<dan> otherwise, we can kill them
<dan> means we don't need to worry about a pseudo-registered state where they're somewhat registered, but can't do a lot of stuff and etc


PBR refractionTexture to default to environmentTexture

As per reflectionTexture in https://github.com/BabylonJS/Babylon.js/pull/2238

I may add a helper for it but we do not want always refraction.
Reflection is a mandatory maybe could do if linkRefractionWithTransparency is true ?

maybe could do if linkRefractionWithTransparency is true ?

Makes sense

Though... I think for the objects I'm rendering what I'd like to do is screen space refraction.
So something like

Render non-transparent (fb1)
swap framebuffer (fb2)
Render transparent refracting using (fb1); outside screen use environment + record depth + stencil
swap back (fb1)
Render (fb2) onto (fb1) using stencil and depth test
Present (fb1)

If that makes sense?

It does and it is kind of what we are doing with reflection probes

http://www.babylonjs.com/demos/refraction/

Kind of:) but I like the idea a lot:)

That's a lot heavier weight? Though, will give it a go :)

It helps when you have curved surface where the retracted pixel is out of screen.
But I also like your screen space idea

Yup I ll do if both are linked together 👍

Done :-)


Vanilla CFR average policy question

Hope asking this via issue is ok as it is only very implicitly related to the implementation in OpenSpiel. I'm reading [1], [2] and [3] and can't seem to find a good explanation about the definition for the average policy. Particularly, I'm wondering why the averate strategy (equation (4) in [1] or equation (3) in [2]) is weighted by player's reach probability at timestamp t (and I see that OpenSpiel implementation does the same)? I realize there's a theorem right after the equations stating the connection betwen the average sigma at T and the equilibrium for zero-sum games but there's no proof for theorem 2 even in appendix of [1].
Is there an intuitive explanation for including reach probabilities? Why wouldn't the cumulative policy simply be the sum of all sigmas over all time steps for a particular action? I'm sure I'm missing something obvious so hopefully this isn't too silly of a question.
References

[1] Zinkevich et al., Regret Minimization in Games with Incomplete Information
[2] Lanctot et al., Monte Carlo Sampling for Regret Minimization in Extensive Games
[3] Neller et al., An Introduction to Counterfactual Regret Minimization

That formula corresponds to computing the average of the sequence-form strategies.
I think the best way to think about the whole process is like this: a Nash equilibrium is a bilinear saddle point problem in the sequence form representation, that is, it can be written as the optimization problem
min_{x \in Q1} max_{y \in Q2} x^T A y

where Q1 is the set of sequence form strategies for Player 1, Q2 is the set of sequence form strategies for Player 2, and A is a sparse payoff matrix.
Now, it turns out that any time you have a bilinear saddle point, if you can build regret minimizers R1, R2 for the domains of the min and max (in this case, the sequence form strategy spaces Q1 and Q2), and you let them play against each other, the average strategies that they produce converge the the the solution of the saddle point problem.
You can think of CFR as a regret minimizer for the set of sequence form strategy of a player. Hence, if both players play according to the CFR regret minimizer, their average sequence form stategy converges to a Nash equilibrium. It is crucial though that you average in sequence form.
The formula with the reaches that you see in those papers is simply taking a behavioral strategy, converting it to sequence form (by multiplying by the reaches), then averaging in sequence form, and finally going back to behavioral representation by dividing by the parent reach of the average sequence form strategy just computed. All in one compact formula.

Thanks @gabrfarina! I need to do some reading in order to better understand why the sequence form is pivotal. Do you have any suggestions for the material? Stengel seems relevant.

@inejc there is a simple example explaining the reaches, slide 33 (from the course about algorithmic game theory). You want to ensure that unreachable parts of the game (right action and then right subtree) do not affect the computation of the average strategy.

I believe I have enough resources to continue the studies. Thank you both.


Simple Examples?

The tutorial leaves a huge amount to be desired, and the sample code attempts to demonstrate every available feature (which is nice if you want to work out how to do advanced stuff).  There is a MASSIVE gap for someone just trying to create a very simpleMDI form with a couple of dockable panels,
Has someone done a simple example?
When I say a simple example, no themes, no dynamic adding of panels, basically as little code as possible to create a Form with 3 panels, a bottom, a right and a main panel that fills the remaining window, where all the panels can be moved and re-parented.

        public Form1()
        {
            InitializeComponent();

            this.IsMdiContainer = true;

            var panel1 = new DockPanel();
            panel1.Dock = DockStyle.Fill;
            Controls.Add(panel1);

            var tab1 = new DockContent();
            tab1.TabText = "My left pane";
            tab1.Show(panel1,DockState.DockLeft);

            var tab2 = new DockContent();
            tab2.TabText = "My mid pane";
            tab2.Show(panel1,DockState.Document);
        }

Hi guys, suppose I've added a number of child controls to the tab1 and tab2 panel, what would be the approach to accessing this child elements (textbox etc...). grateful


fix(Meeting attendance): update some colors for weekend meeting

:tada: This PR is included in version 2.130.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


⚠ Bofrost.at has degraded performance

In a3c0fd4, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1302 ms

Resolved: Bofrost.at performance has improved in 419aaa9 after 11 minutes.


Promote HAControlPlanes feature gate to beta

How to categorize this PR?

/area high-availability
/kind enhancement
What this PR does / why we need it:
Promote HAControlPlanes feature gate to beta and turn it on by default.
Which issue(s) this PR fixes:
Part of #6529
Release note:

The `HAControlPlanes` feature gate has been promoted to beta and is now turned on by default.

/hold until<EMAIL_ADDRESS>got released

/approve

/unhold
since<EMAIL_ADDRESS>is released


Fixed bug with image linting and added additional options to loc-scop…

Fixed bug with image linting and added additional options to loc-scope image extension.

Referencing Open Issue #282


CoinMarketCap API v1 will be decommissioned

Component/platform:
https://www.home-assistant.io/components/sensor.coinmarketcap/
Description of problem:
CoinMarketCap Public API Version 1 will be taken offline on November 30th, 2018.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
sensor:
  - platform: coinmarketcap

This should be resolved first: https://github.com/barnumbirr/coinmarketcap/issues/40


Ensure alt-texts ends with punctuations

Fixes #2814
I edited the script to include blocks that will only add full stops to alt-texts if they do not have a full stop, question or exclamation mark.

@AfiDede would you like to add yourself to the CONTRIBUTORS.yaml file in the repository? It would help recognise your contributions to the project :)

Yes Helena I will be doing that. Thank you !
On Wed, Oct 20, 2021 at 7:40 AM Helena @.***> wrote:

@AfiDede https://github.com/AfiDede would you like to add yourself to
the CONTRIBUTORS.yaml file in the repository? It would help recognise your
contributions to the project :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/galaxyproject/training-material/pull/2875#issuecomment-947409365,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APBCR7B4KHZTYX62NFJO6PTUHZW6RANCNFSM5GHFUAFQ
.


Modified logger configuration in init.py (libraries shouldn't modify root logger)

Very minor tweak suggested to__init__.py only.
I've been getting duplicate log messages when I import txtai into my code.  It turns out txtai modifies Python's root logger in it's __init__.py using logging.basicConfig().  However it is recommended that "importable" libraries do not configure log handlers/formatters, and that they should leave that to the "application" code.
So I've edited this line to better reflect the recommendations from the official python docs.

Sorry to hear you are getting duplicate log messages. Thank you for the fix!
While I appreciate the thorough explanation, I prefer to keep these types of longer comments in GH issues/commit logs. Otherwise, the code can get messy over time.
Can you make the following minor condensation of the comment and I'll go ahead and merge this in.
# Configure logging per standard Python library recommendations
logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

I've committed the tweak suggested.  Thanks for the rapid response. 👍🏼
I should've said if it turns out that the log messages are useful for tests/debugging (I imagine they are), then I guess those tests should be considered as "application" code and should be setting up their own log handlers etc where needed.  I'm still getting comfortable with logging best practises in Python but I understand that my change above brings txtai in line with how users expect importable packages to behave w.r.t. logging.

All sounds good and makes sense that txtai shouldn't change logging behavior by default. Thanks again.


Migration to Svelte 5

Most functionalities stays the same as of version 0.5.6, except for all $ usage for reactivity will be replaced with Svelte Runes.
Documentaiton: https://svelte.dev/docs/svelte/what-are-runes

This is an unexpected priority due to the documentation for Svelte 4 is now legacy and kinda "gone for good". For the sustainability of this project, I think it's better to spend some time doing the migration (and potentially participate in SvelteHack)

Documentation for the current code: https://v4.svelte.dev/docs/introduction


Create outputDirectory if not existing

If the outputDirectory is not existing on disk, it won't be created. It would be neat, if the the directory would be created and the build command would be triggered.
[Info  - 12:12:59 AM] Non-fatal error loading modules: Couldn't locate your output directory at: /home/andreas/dev/project/ls-output
[Info  - 12:12:59 AM] Output directory does not exist at '/home/andreas/dev/project/ls-output'
[Error  - 12:13:21 AM] Error message from IDE server reloading modules: Couldn't locate your output directory at: /home/andreas/dev/project/ls-output

Just creating an empty output directory would start the server with no module information. A missing output directory means that either the server is started in the wrong directory or no full build has taken place yet, both of which mean the server can't be functional after it's started.

I think what purs-loader does, if you configure an output directory, is creating the directory, and build its content.
It would be nice if purescript-language-server would behave similar.

I think what purs-loader does, if you configure an output directory, is creating the directory, and build its content

How does it do that? Does it own the build process? The LS doesn't. It works on projects built with spago, pulp, nix, or your own custom solution so

and build its content

is just not something it knowns how to do.

I'm with you that it might not be the responsibility of the language server. But I'm somewhat confused, because code looks like building the project is something it can do.

building the whole project is just not something it knowns how to do.

https://github.com/nwolverson/purescript-language-server/blob/5b77635ac9dfbeced72c779d14628674395a23df/src/LanguageServer/IdePurescript/Main.purs#L335-L339
https://github.com/nwolverson/purescript-language-server/blob/63d2c9160da2f6c71f7cf3a061df720b46a21a49/src/IdePurescript/Build.purs#L56-L89

But I'm somewhat confused, because code looks like building the project is something it can do.

Oh! I thought that would be in the editor specific plugins. Allright let's think about this again then... I fear for the UX here, but let's think about a couple of scenarios

Running the build command might fail because there's an actual error in the project

In that situation you sometimes still want to start the LS because it can use the information of all the modules that did compile to help you fix these errors. But sometimes this is an indication of a bigger problem like version mismatches that you really want the user to know about and fix before continuing. So we'd need to be very smart or very chatty to not lead users astray here.

Running the build without prompting the user on startup might fail because the LS guessed the build command wrong

How do we tell his apart from 1.? Horror scenario would be to guess the build command wrong but still run to completion and get the wrong output, but that seems very unlikely at least.

Running the build command takes a long time.

It's not uncommon for a completely fresh clone of a large PS project to take a minute or two to compile from scratch (especially if you're also downloading the packages). Will this make editors assume the language server timed out? This would probably need a progress bar/indicator, but I don't think spago makes it easy to extract that progress (doesn't have an API for programs calling it).
This is what I can come up with right now... the problems all seem solvable but sound like some thought and care is needed to get this right.

Unless I'm mistaken, @andys8 you've just indicated that when you start without output directory existing, the option to build project is presented?
That seems like an ideal solution, for the reasons that @kritzcreek gave I wouldn't want to trigger this automatically - some of the time this will be the situation for setup reasons rather than just needing to build, and the build command may not be correctly configured - but here we give the user information, and make it easy to build and proceed. I'm concerned that firing this off automatically will just lead to less obvious support issues if anything.
Is the issue that this functionality didn't work for you? It sounds that you were unaware of it in the original issue description. I certainly wouldn't be surprised many LSP clients do not implement the action button functionality on warning messages.

I try to improve the quality of my comments regarding this issue. Initially, I tried to set the outputDirectory and noted, it's just not working.
I went with a different way to solve this for my project, but wanted to create an issue, so this is tracked. The initial issue was missing details.
I also think it's less critical than e.g. #85 where code actions are not working in Vim.
Starting with more clarification:

The issue is meant to be a bug report and not a feature request
The build server already prompts with an action to trigger a build
I wouldn't do this automatically instead, too

I initially though, the build is not triggered/working at all. But (see 2.) it is.
The language server / my configuration wasn't using the correct outputDirectory in the build command.
I guess I didn't notice, because the output (wrong directory) was still existing in my past tests.
Tests
1. No language server config
Clean repo with no output folder

Prompts with build
Compiles successfully

[Info  - 6:15:07 PM] Starting IDE server on port 15721 with cwd /home/andreas/dev/ebot7/conversation-ide-poc
Started IDE server (port 15721)
[Info  - 6:15:07 PM] Your output directory didn't exist. This usually means you didn't compile your project yet.
psc-ide needs you to compile your project (for example by running pulp build)
[Info  - 6:15:07 PM] Non-fatal error loading modules: Couldn't locate your output directory at: /home/andreas/dev/ebot7/conversation-ide-poc/output
[Info  - 6:15:07 PM] Output directory does not exist at '/home/andreas/dev/ebot7/conversation-ide-poc/output'
[Info  - 6:15:14 PM] Resolved build command (1st is used):
[Info  - 6:15:14 PM] /home/andreas/.npm-global/bin/spago: 0.16.0
[Info  - 6:15:14 PM] Running build command: spago build --purs-args --json-errors
...
Compiling Main
...
[Info  - 6:15:39 PM] [info] Build succeeded.
[Info  - 6:15:39 PM] Build complete

### 2. Configured `outputDirectory`

Clean repo with no `output` folder

"outputDirectory": "other-directory/"

* Prompts with build
* spago builds to `output`
* Therefore not working

### 3. Trying to configure `buildCommand`

[WIP] With the results of 2, I think the cause is likely that the `buildCommand` needs to be adapted.

"outputDirectory": "other-directory/",
"buildCommand": "spago build --purs-args "-o other-directory --json-errors""

I must be doing something wrong with escaping here.

[Info  - 6:34:33 PM] Running build command: spago build --purs-args ""-o other-directory --json-errors""
[Info  - 6:34:33 PM] Invalid argument `other-directory'

So my current understanding is, that the language server implementation would work, if the `buildCommand` would be adapted correctly.


Vertices not updating

Hi, 👋
first of all I really enjoy playing around with react-three-fiber and I've even started to write my own thin wrapper around another library using react-three-fiber's reconciler 🎉.
While playing around with react-three-fiber I've found that my vertices are not updating, that might be an Three.js issue.
Here's a quick example, the button 'Add vertex' dynamically adds a new Vertex on each click, and applyProps gets properly called, but in the end the vertices are not updated: https://codesandbox.io/s/qqq7qpqvo9

Looks odd indeed. I logged the geometry, the vectors are in there. Mutating works, too: https://codesandbox.io/s/m74413rlq8 Could be that the size of the vertices isn't supposed to change in three?

Sorry, looks like it's confirmed: https://stackoverflow.com/questions/14840026/dynamically-adding-vertices-to-a-line-in-three-js

You can only update content of buffers, you cannot resize buffers (this is very costly, basically equivalent to creating new geometry).
You can emulate resizing by pre-allocating a larger buffer and then keeping unneeded vertices collapsed / hidden.

It's interesting, because previously i wouldn't even have tried to re-use objects, one of the reasons i wanted a reconciler.

@drcmda thanks for looking into this, I think it can be closed then.
I've seen that you closed this issue and then reopened shortly after, was the reopening a mistake?

yep, clicked the wrong button, just didn't want to close preemptively.


Segmentation Violation

Describe the bug
When running any type of scan dalfox throws a segmentation violation
Environment

Dalfox Version: 2.5.2
Installed from: go-get

Hi @Nimbus689
First, thank you for submit issue!
I checked the part in the code that could cause the error, although the error did not occur during the my test case. (If the server blocks or dies during the scan)
I added defense logic in dev branch. can you check?
Test with dev branch
$ git clone https://github.com/hahwul/dalfox
$ git branch dev
$ git checkout dev
$ git pull -v
$ go build
$ ./dalfox

I ran the dev branch code and it is still happening. It was working before I upgraded to 2.5.0.... I was able to check out 2.5.2 on a different machine and it runs as expected there must be something broken when I updated dalfox

@Nimbus689
Yes, WAF detection and evasion have been added to v2.5.0, and it seems to be a problem that occurs in that area.
It would be great if it could be reproduced in my environment, but I don't have any errors as below 😭
Ubuntu

Kali (docker)

Hm... I'll test it a bit more! It is a little strange that the problem occurs even though the write to exception.

It's alright don't waste anymore time on it. I've been meaning to swap my machine over to a new one anyway so I'll just go ahead and do it :)
Thanks for being responsive to the tickets and for making a great tool!

I'm actually seeing this same error on a lot of URL's  after the upgrade. It's not happening on all URL's but I have automation that batch in a file of 100 at a time and it is throwing this error consistently before it finishes a batch. Would more URL's to test with help?
[I] Found 15 testing point in DOM base parameter mining
⠼  [SA: ✓ PA: ✓ BAV: ✓ ] Waiting for analysis 🔍 panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xa14eee]
goroutine 1745 [running]:
github.com/hahwul/dalfox/v2/pkg/scanning.ParameterAnalysis.func7(0xc000b1a000, 0xc000b7cb00, 0x7ffe64dcf3b1, 0x2c, 0xc000676040, 0xc001240020, 0xc001240018, 0xc000678020, 0xc000b460c0, 0xc001240010, ...)
/home/panch0r3d/go/pkg/mod/github.com/hahwul/dalfox/v2@v2.5.2/pkg/scanning/parameterAnlaysis.go:202 +0x40e
created by github.com/hahwul/dalfox/v2/pkg/scanning.ParameterAnalysis
/home/panch0r3d/go/pkg/mod/github.com/hahwul/dalfox/v2@v2.5.2/pkg/scanning/parameterAnlaysis.go:194 +0x3d3

@panch0r3d cc) @Nimbus689
Oh, I think that's a big problem. Is there any way, can you test the 2.5.3-dev version?
2.5.3-dev (dev branch)
$ git clone https://github.com/hahwul/dalfox
$ git branch dev
$ git checkout dev
$ git pull -v
$ go build
$ ./dalfox

@Nimbus689
When i told you for setup dev branch command, the pull might not be right. Can you check again to see if it's possible with the command below?
git clone https://github.com/hahwul/dalfox
git branch dev
git checkout dev
git pull origin dev -v
go build
./dalfox

@panch0r3d @Nimbus689
If it is fixed with dev branch code, I will proceed with hotfix release(v2.5.3) right away!
(It's frustrating that I can't trigger problem in my machine😵)

@panch0r3d @Nimbus689
First off, I'll do hotfix process right away! If there's any issue after 2.5.3, please let me know!
I'm really sorry for causing you discomfort due to code problems! I wasn't very meticulous 😱
Anyway, thank you so much for great help in solving this issue! Enjoy your week 🥰

Hey I know it is closed already but I wanted to confirm that the issue has been resolved for me with the 2.5.3 release. So thanks again :D


libarch_powerpc.so causing failure to start binaryninja on Ubuntu 18.04

Installing on a base Ubuntu 18.04 I was presented with the following error when attempting to start binaryninja:
./binaryninja -d -e Using ca bundle: /etc/ssl/certs/ca-certificates.crt : CommandLine Error: Option 'asm-instrumentation' registered more than once! LLVM ERROR: inconsistency in registered CommandLine options
This is a fatal error and binaryninja does not start.
Upon investigation I discovered that the error was with libarch_powerpc.so. By removing this file from the plugin directory I was able to get binaryninja to start without error.

Fixed in build 1.1.1231-dev.


Perform NULL check for "except +" calls

This is an attempt to fix #2603.  I'm not very familiar with Cython's code - this is my first time really looking at the code generator at all - so please let me know if there's anything that needs refinement.  I've additionally added the test case from #2603 to the test suite.

Thanks for looking at this!
The only comment I have is that there are several other places that need similar logic (see the uses of translate_cpp_exception). I'll merge what you have into mine and push upstream to close this bug.


Update docfx.json add custom searchScopes for CLI

Repairing metadata for ms.service for Redis cache
Adding a search scope so that Azure CLI searches are unified from the TOC filter.
Adding custom search scopes for big data services so that, and the service-specific searches also work across all docsets, and Azure docs all up can be searched.

Docs Build status updates of commit 39537b9:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

Line 220: [Error] Invalid syntax in W:\nnfd-s\docfx.json. Please validate your syntax with a JSON validator like http://jsonlint.com/ and update the JSON file in the repo with correct syntax.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 12f0c5c:
:clock10: Full build: the average full build time is 7 min(s) 6 sec(s), this is based on the last 267 build(s) for this repository.

Docs Build status updates of commit 12f0c5c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

In the review site build, testing the general case for the search scope:

Checking the review build
https://review.docs.microsoft.com/en-us/cli/azure/redis?view=azure-cli-latest&branch=pr-en-us-1661&viewFallbackFrom=azure-cli-2017-03-09-profile#az-cache-delete
<meta name="ms.service" content="cache">
<meta name="scope" content="Azure, Azure CLI" />

#sign-off
I don't think the local folder search scopes are working but I will be OOF and will address that later.

#sign-off @sdwheeler

Docs Build status updates of commit 14affbd:
:clock10: Full build: the average full build time is 7 min(s) 6 sec(s), this is based on the last 269 build(s) for this repository.

Docs Build status updates of commit 14affbd:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

2017-03-09-profile/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/index.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/maintenance.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/maintenance/applyupdate.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/maintenance/assignment.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/maintenance/configuration.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/maintenance/maintenance/update.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2018-03-01-hybrid/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/index.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/applyupdate.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/assignment.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/configuration.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2018-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/update.yml
:white_check_mark:Succeeded
View (azure-cli-2018-03-01-hybrid)

2019-03-01-hybrid/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/index.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/applyupdate.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/assignment.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/configuration.yml
:white_check_mark:Succeeded

2019-03-01-hybrid/docs-ref-autogen/ext/maintenance/maintenance/update.yml
:white_check_mark:Succeeded

docfx.json
:white_check_mark:Succeeded

latest/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/acr/scope-map.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/acr/token.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ams/asset.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/index.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/maintenance.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/maintenance/applyupdate.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/maintenance/assignment.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/maintenance/configuration.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/maintenance/maintenance/update.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


where is the train & test code ?

Hi, thanks for nice research and code upload.
I viewed the repository, but couldn't find the main train & estimation code.
where is it ? I'm sorry If I missed something.

Hi, yes you are right. Currently, the repo only contains dataset related code. The Fusion Network is still being refactored and is in progress.

HI, @MarioBijelic , any update news on the release of training and testing code?
THX!


Can't use externalized syntax highlighting

Oni Version: 0.3.6
Neovim Version (Linux only): 0.3.1
Operating System: Manjaro 17.1.12, Linux 4.14.66-1-MANJARO x86_64
Issue: Can't use externalized syntax highlighting
Expected behavior: When I want to use a syntax highlighting provided by some Vim plugin and I disable TextMate syntax, the one coming from the plugin should be used
Actual behavior: Default syntax is shown
Steps to reproduce:
Use SemanticHighlight plugin:
syntax on
Plug 'jaxbot/semantic-highlight.vim'

Disable TextMate:
"editor.textMateHighlighting.enabled": false,
"oni.loadInitVim": true,

Open a file and launch SemanticHighlight:
:SemanticHighlight
In nvim:

In oni:

I think this may be due to how the plugin checks if a GUI is running.
The plugin checks the following to check if it should use GUI colours:
https://github.com/jaxbot/semantic-highlight.vim/blob/01e646ca17dfe0143ef9c8a04a684e8f100bd20d/plugin/semhl.vim#L144
Of which none are set in Oni, due to either not making sense to enable or not existing to enable.
If I manually force the plugin to work in Oni (by adding an exists('g:gui_oni') to the check above), then the plugin works as expected:

However, that is obviously not a fix at all.
I'm not sure what the general nvim way of doing this is, except for checking nvim_list_uis() and checking the RGB flag it returns (which Oni does set). Perhaps someone more versed in how to do this could jump in and say?
But after a change is made to the plugin, the setup you've got should work as expected. Its maybe a different issue of if Oni should stop the textmate highlights if it detects some new user enabled ones.

You are saying that the best way is to make SemanticHighlight compatible to oni, right? I think that since there are two plugins that don't work because of oni, maybe oni itself could try to make an effort about this issue... But I'm not a Vim developer, so maybe I'm wrong.

You are saying that the best way is to make SemanticHighlight compatible to oni, right?

Not quite, more just to be compatible with any Neovim GUI, not just Oni. Ie, the current approach means it also doesn't work in nvim-qt, NyaoVim and I assume any other neovim gui that uses GUI colours.
There is a PR that would fix it for some of them (https://github.com/jaxbot/semantic-highlight.vim/pull/61) but it sounds like setting &termiguicolors in Oni isn't really appropriate, as per @justinmk's comment on the Goyo issue.
Using nvim_list_uis() seems to be the way to work for all nvim GUIs from now on, not just Oni. I think since the nvim GUIs (including Oni!) are still mainly fairly early on, some plugins don't react to them very well yet, and also why not many plugins have adapted to them.

Ok, for now I have added support to Oni in my fork version of SemanticHighlight. I plan to come back and implement the nvim_list_uis() way.

Shouldn't we make gui_running be true on Oni?

Shouldn't we make gui_running be true on Oni?

I don't know if that exists in the nvim side of things. I did a :helpgrep gui_running and the only reference I found to it was in the ALE config. If it does, then possibly, but my very basic understanding is that it relates to GVim rather than a GUI in general?

If it does, then possibly, but my very basic understanding is that it relates to GVim rather than a GUI in general?

Yes, also Macvim.
However it seems that this is the case because they were the only GUIs available at the time. I think it would be sensible to use this as many plugins may use this variable to detect capabilities, e.g. instead of using t_Co detecting if it is a GUI, etc..

@justinmk Thanks for the feedback!