Fail to compile the example in the repository: Configuration is not in the right format. FormatExceptions: [FormatException: No value was provided for required key: package_name

Hi thanks for the toolchain! However I fail to compile the example. Reproduction: git clone, and cd into pkgs/native_assets_cli/example/native_add_library Execute dart --enable-experiment=native-assets test and get:

(base) ➜  native_add_library git:(main) dart --enable-experiment=native-assets test
Unhandled exception:
FormatException: Configuration is not in the right format. FormatExceptions: [FormatException: No value was provided for required key: package_name, #0      Config._throwIfNull (package:cli_config/src/config.dart:591:7)
#1      Config.string (package:cli_config/src/config.dart:213:5)
#2      BuildConfig._readFieldsFromConfig.<anonymous closure> (package:native_assets_cli/src/model/build_config.dart:321:41)
#3      new BuildConfig.fromConfig (package:native_assets_cli/src/model/build_config.dart:249:10)
#4      BuildConfig.fromArgs (package:native_assets_cli/src/model/build_config.dart:286:24)
<asynchronous suspension>
#5      main (file:///Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/build.dart:15:23)
<asynchronous suspension>
]
#0      new BuildConfig.fromConfig (package:native_assets_cli/src/model/build_config.dart:257:7)
#1      BuildConfig.fromArgs (package:native_assets_cli/src/model/build_config.dart:286:24)
<asynchronous suspension>
#2      main (file:///Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/build.dart:15:23)
<asynchronous suspension>
Building native assets for package:native_add_library failed.
build.dart returned with exit code: 255.
To reproduce run:
/Users/tom/fvm/versions/3.16.0/bin/cache/dart-sdk/bin/dart --packages=/Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/.dart_tool/package_config.json /Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/build.dart --config=/Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/.dart_tool/native_assets_builder/9fe3bb4141b182a788cfb7907563ade2/config.yaml
stderr:
Unhandled exception:
FormatException: Configuration is not in the right format. FormatExceptions: [FormatException: No value was provided for required key: package_name, #0      Config._throwIfNull (package:cli_config/src/config.dart:591:7)
#1      Config.string (package:cli_config/src/config.dart:213:5)
#2      BuildConfig._readFieldsFromConfig.<anonymous closure> (package:native_assets_cli/src/model/build_config.dart:321:41)
#3      new BuildConfig.fromConfig (package:native_assets_cli/src/model/build_config.dart:249:10)
#4      BuildConfig.fromArgs (package:native_assets_cli/src/model/build_config.dart:286:24)
<asynchronous suspension>
#5      main (file:///Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/build.dart:15:23)
<asynchronous suspension>
]
#0      new BuildConfig.fromConfig (package:native_assets_cli/src/model/build_config.dart:257:7)
#1      BuildConfig.fromArgs (package:native_assets_cli/src/model/build_config.dart:286:24)
<asynchronous suspension>
#2      main (file:///Volumes/MyExternal/ExternalRefCode/native/pkgs/native_assets_cli/example/native_add_library/build.dart:15:23)
<asynchronous suspension>

stdout:

Error: Compiling native assets failed.

Then I realized it is because v0.3.0 has a breaking change. Thus I make this issue, such that people who is in the same situation as me (using the stable Dart 3.2 + the in-tree example, coming from https://dart.dev/guides/libraries/c-interop#native-assets webpage), can search and find this page and avoid wasting a few minutes finding out manually why :)
By the way, maybe it is better to add a hint in https://dart.dev/guides/libraries/c-interop#native-assets to hint users about this?

While this feature is experimental, we might need to break the protocol between the Dart/Flutter SDK and the build.dart scripts a couple of more times while we iron out the protocol.
Our goal is to bump the protocol version number when making such changes.
https://github.com/dart-lang/native/blob/893433ee9e7a5bfbc8ebb52840c9448bda124f2c/pkgs/native_assets_cli/lib/src/model/build_config.dart#L242
However, I think have forgotten to do that in https://github.com/dart-lang/native/pull/150. Sorry for the confusion! 🙊 With that version being bumped there should be a clearer error message.


cannot import name 'GPTSimpleVectorIndex' from 'llama_index'

Hi,
I'm having trouble importing the GPTSimpleVectorIndex class from the llama_index package. I've installed the package using pip, but I'm getting an ImportError when trying to import the class. Is there any known issue or workaround for this problem? I've followed the documentation, but I can't seem to resolve the ImportError.
Used :
pip install llama-index
from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, Document, GPTSimpleVectorIndex

GPTSimpleVectorIndex was renamed to GPTVectorStoreIndex, try removing it from the end of your imports

@logan-markewich Thanks Logan. Its working now. However, do you know if there is a way I can summarise the texts and display the response using Llama Index? I mean at the moment, the code base is extracting the information from the files and displaying it word to word.

Have you followed the basic usage guide? It will definitely explain a lot:
https://gpt-index.readthedocs.io/en/stable/guides/primer/usage_pattern.html
At a basic level, llama index takes your documents and breaks them into chunks/nodes.
GPTVectorStoreIndex will embed each node, and then at query time, use your query text to fetch the top k most relevant nodes and use that as context to synthesize an answer with the LLM
For summaries, you can use either GPTListIndex or GPTVectorStoreIndex with response_mode="tree_summarize". The difference is, a list index will use every node in your index to write the summary, where the vector index will use only the top k nodes to write a summary

Sure, @logan-markewich, I will give it a try. Thanks

@logan-markewich The index created via GPTSimpleVectorIndex had a save_to_disk('xxxx.json') method. Is there a replacement plan, or what's your recommendation?

Have you followed the basic usage guide? It will definitely explain a lot: https://gpt-index.readthedocs.io/en/stable/guides/primer/usage_pattern.html
At a basic level, llama index takes your documents and breaks them into chunks/nodes.
GPTVectorStoreIndex will embed each node, and then at query time, use your query text to fetch the top k most relevant nodes and use that as context to synthesize an answer with the LLM
For summaries, you can use either GPTListIndex or GPTVectorStoreIndex with response_mode="tree_summarize". The difference is, a list index will use every node in your index to write the summary, where the vector index will use only the top k nodes to write a summary

when trying to add response_mode="compact" it says it doesn't accept it anymore.
and cant find any example related to the repsonse_mode in the documentation
any ideas?

Response mode goes into the query engine now
query_engine = davinci_index.as_query_engine(
    response_mode="compact"
)
response = query_engine.query(...)

Initially I used index = GPTSimpleVectorIndex.load_from_disk('index.json')    How to fix this with GPTVectorStoreIndex? Which function from GPTVectorStoreIndex I can use inplace of load_from_disk?

Initially I used index = GPTSimpleVectorIndex.load_from_disk('index.json') How to fix this with GPTVectorStoreIndex? Which function from GPTVectorStoreIndex I can use inplace of load_from_disk?      and how to fix this also
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context).                                                The complete code is as follows.
from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper, ServiceContext
from langchain import OpenAI
import sys
import os
from IPython.display import Markdown, display
def construct_index(directory_path):
# set maximum input size
max_input_size = 4096
# set number of output tokens
num_outputs = 2000
# set maximum chunk overlap
max_chunk_overlap = 20
# set chunk size limit
chunk_size_limit = 600
# define prompt helper
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

# define LLM
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0.5, model_name="text-davinci-003", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)

index.save_to_disk('index.json')

return index

def ask_ai():
try:
index = GPTSimpleVectorIndex.load_from_disk('index.json')
while True:
query = input("What do you want to ask? ")
response = index.query(query)
display(Markdown(f"Response: {response.response}"))
except Exception as e:
print(f"An error occurred: {e}")

AttributeError: 'GPTVectorStoreIndex' object has no attribute 'save_to_disk'

query_engine = index.as_query_engine(
response_mode="compact"
)

Thanks it worked

Initially I used index = GPTSimpleVectorIndex.load_from_disk('index.json') How to fix this with GPTVectorStoreIndex? Which function from GPTVectorStoreIndex I can use inplace of load_from_disk? and how to fix this also index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context). The complete code is as follows.
from llama_index import SimpleDirectoryReader, GPTListIndex, readers, GPTSimpleVectorIndex, LLMPredictor, PromptHelper, ServiceContext from langchain import OpenAI import sys import os from IPython.display import Markdown, display
def construct_index(directory_path): # set maximum input size max_input_size = 4096 # set number of output tokens num_outputs = 2000 # set maximum chunk overlap max_chunk_overlap = 20 # set chunk size limit chunk_size_limit = 600
# define prompt helper
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

# define LLM
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0.5, model_name="text-davinci-003", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)

index.save_to_disk('index.json')

return index

def ask_ai(): try: index = GPTSimpleVectorIndex.load_from_disk('index.json') while True: query = input("What do you want to ask? ") response = index.query(query) display(Markdown(f"Response: {response.response}")) except Exception as e: print(f"An error occurred: {e}")

To construct your index use :
    # set maximum input size
    max_input_size = 4096
    # set number of output tokens
    num_outputs = 256
    # set maximum chunk overlap
    max_chunk_overlap = 20
    # set chunk size limit
    chunk_size_limit = 600

    # define LLM (ChatGPT gpt-3.5-turbo)
    llm_predictor = LLMPredictor(llm=AzureChatOpenAI(deployment_name="gpt-35-turbo", temperature=0.3,openai_api_version=openai_api_version))
    prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
    embeddings = LangchainEmbedding(OpenAIEmbeddings(model="text-embedding-ada-002", chunk_size=1, openai_api_version=openai_api_version))

    documents = SimpleDirectoryReader(directory_path).load_data()

    service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper,embed_model=embeddings)
    index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context, prompt_helper=prompt_helper)

    index.storage_context.persist(persist_dir="<persist_dir>")
    return index

To load  your index back use and use it:
    storage_context = StorageContext.from_defaults(persist_dir="<persist_dir>")

    # load index
    index = load_index_from_storage(storage_context)

    query_engine = index.as_query_engine(service_context=service_context, verbose=True,response_mode="compact")

    template = """
    You are a helpful university chatbot that gets asked questions from students. You are given many questions and answer and your goal is to take the student question and check the most similar question available and use the answer. Answer with the same language as the question: The question is : {text} and the answer is:
    """
    mod_question = template.format(text=question)
    # mod_question = question
    answer = query_engine.query(mod_question)```

I am currently working with a few PDFs and DOCX files located in the /content folder in Colab. These files are being indexed and I am able to get a response. However, I would like to obtain the file path or the file itself from the response and print the respective file in a downloadable format (.docx or .pdf). Is there a way to achieve this?

load index
index = load_index_from_storage(storage_context)

Where does the function load_index_from_storage come from?

AttributeError: 'GPTVectorStoreIndex' object has no attribute 'save_to_disk'

Change in the 1st line -> from llama_index import SimpleDirectoryReader, GPTListIndex, GPTVectorStoreIndex, LLMPredictor, PromptHelper, ServiceContext, StorageContext, load_index_from_storage
#Please use the following to store information on the disk
index = GPTVectorStoreIndex.from_documents(documents)

index.set_index_id("vector_index")

store on the disk
index.storage_context.persist('storage')

For retrieval plase use the method as follows.
def chatbot(input_text):
# rebuild storage context
storage_context = StorageContext.from_defaults(persist_dir='storage')
# load index
index = load_index_from_storage(storage_context, index_id="vector_index")
query_engine = index.as_query_engine()

response = query_engine.query(input_text)

#response = index.query(input_text, response_mode="compact")
return response.response

I have this code and it is giving me a GPTSimpleVectorIndex error. Also, suggest the changes that I need to make regarding the error. Thanks
import os
import sys
from docx import Document
from langchain import OpenAI
from llama_index import SimpleDirectoryReader, LangchainEmbedding, GPTListIndex, GPTSimpleVectorIndex, PromptHelper
from llama_index import LLMPredictor, ServiceContext
os.environ["OPENAI_API_KEY"] = '..'
def convert_docx_to_txt(docx_file_path, txt_file_path):
document = Document(docx_file_path)
with open(txt_file_path, 'w', encoding='utf-8') as txt_file:
for paragraph in document.paragraphs:
txt_file.write(paragraph.text + '\n')
def construct_index(directory_path):
max_input_size = 4096
num_outputs = 256
max_chunk_overlap = 20
chunk_size_limit = 600
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

# Change the model_name parameter to "text-davinci-003"
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)

index.save_to_disk('index.json')

return index

def ask_bot(query, input_index='index.json'):
index = GPTSimpleVectorIndex.load_from_disk(input_index)
response = index.query(query, response_mode="compact")
return response
def main():
docx_file_path = "your_word_document.docx"
txt_file_path = "converted_text.txt"
if not os.path.exists(txt_file_path):
convert_docx_to_txt(docx_file_path, txt_file_path)
if not os.path.exists('index.json'):
    index = construct_index('./')  # Use the current directory for simplicity

ask_bot('index.json')

I have this code and it is giving me a GPTSimpleVectorIndex error. Also, suggest the changes that I need to make regarding the error. Thanks
import os import sys from docx import Document from langchain import OpenAI from llama_index import SimpleDirectoryReader, LangchainEmbedding, GPTListIndex, GPTSimpleVectorIndex, PromptHelper from llama_index import LLMPredictor, ServiceContext
os.environ["OPENAI_API_KEY"] = '..'
def convert_docx_to_txt(docx_file_path, txt_file_path): document = Document(docx_file_path) with open(txt_file_path, 'w', encoding='utf-8') as txt_file: for paragraph in document.paragraphs: txt_file.write(paragraph.text + '\n')
def construct_index(directory_path): max_input_size = 4096 num_outputs = 256 max_chunk_overlap = 20 chunk_size_limit = 600
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

# Change the model_name parameter to "text-davinci-003"
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003", max_tokens=num_outputs))

documents = SimpleDirectoryReader(directory_path).load_data()

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)

index.save_to_disk('index.json')

return index

def ask_bot(query, input_index='index.json'): index = GPTSimpleVectorIndex.load_from_disk(input_index) response = index.query(query, response_mode="compact") return response
def main(): docx_file_path = "your_word_document.docx" txt_file_path = "converted_text.txt" if not os.path.exists(txt_file_path): convert_docx_to_txt(docx_file_path, txt_file_path)
if not os.path.exists('index.json'):
    index = construct_index('./')  # Use the current directory for simplicity

ask_bot('index.json')

I have this code and it is giving me a GPTSimpleVectorIndex error. Also, suggest the changes that I need to make regarding the error. Thanks
import os
import sys
from docx import Document
from langchain import OpenAI
from llama_index import SimpleDirectoryReader, LangchainEmbedding, GPTListIndex, GPTSimpleVectorIndex, PromptHelper
from llama_index import LLMPredictor, ServiceContext
os.environ["OPENAI_API_KEY"] = '..'
def convert_docx_to_txt(docx_file_path, txt_file_path):
document = Document(docx_file_path)
with open(txt_file_path, 'w', encoding='utf-8') as txt_file:
for paragraph in document.paragraphs:
txt_file.write(paragraph.text + '\n')
def construct_index(directory_path):
max_input_size = 4096
num_outputs = 256
max_chunk_overlap = 20
chunk_size_limit = 600
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
Change the model_name parameter to "text-davinci-003"
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003", max_tokens=num_outputs))
documents = SimpleDirectoryReader(directory_path).load_data()
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)
index.save_to_disk('index.json')
return index
def ask_bot(query, input_index='index.json'):
index = GPTSimpleVectorIndex.load_from_disk(input_index)
response = index.query(query, response_mode="compact")
return response
def main():
docx_file_path = "your_word_document.docx"
txt_file_path = "converted_text.txt"
if not os.path.exists(txt_file_path):
convert_docx_to_txt(docx_file_path, txt_file_path)
if not os.path.exists('index.json'):
index = construct_index('./')  # Use the current directory for simplicity
ask_bot('index.json')

def createVectorIndex(path):
max_input = 4096
tokens = 200
chunk_size = 600 #for LLM, we need to define chunk size
max_chunk_overlap = 20
#define prompt
promptHelper = PromptHelper(max_input,tokens,chunk_overlap_ratio=0.2,chunk_size_limit=chunk_size)

#define LLM — there could be many models we can use, but in this example, let’s go with OpenAI model
llmPredictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-ada-001",max_tokens=tokens))

#load data — it will take all the .txtx files, if there are more than 1
docs = SimpleDirectoryReader(path).load_data()

vectorIndex = GPTVectorStoreIndex(documents=docs,llm_predictor=llmPredictor,prompt_helper=promptHelper)

vectorIndex.save_to_disk('vectorIndex.json')

return vectorIndex

vectorIndex = createVectorIndex('d:\channel\knowledge')

ValueError                                Traceback (most recent call last)
c:\Users\Dell\OneDrive\Desktop\python\programs\test.ipynb Cell 5 in ()
----> 1 vectorIndex = createVectorIndex('d:\channel\knowledge')
c:\Users\Dell\OneDrive\Desktop\python\programs\test.ipynb Cell 5 in createVectorIndex(path)
13 #load data — it will take all the .txtx files, if there are more than 1
14 docs = SimpleDirectoryReader(path).load_data()
---> 17 vectorIndex = GPTVectorStoreIndex(documents=docs, llm_predictor=llmPredictor, prompt_helper=promptHelper)
19 vectorIndex.save_to_disk('vectorIndex.json')
21 return vectorIndex
File c:\Users\Dell\anaconda3\lib\site-packages\llama_index\indices\vector_store\base.py:46, in VectorStoreIndex.init(self, nodes, index_struct, service_context, storage_context, use_async, store_nodes_override, show_progress, **kwargs)
44 self._use_async = use_async
45 self._store_nodes_override = store_nodes_override
---> 46 super().init(
47     nodes=nodes,
48     index_struct=index_struct,
49     service_context=service_context,
50     storage_context=storage_context,
51     show_progress=show_progress,
52     **kwargs,
53 )
File c:\Users\Dell\anaconda3\lib\site-packages\llama_index\indices\base.py:47, in BaseIndex.init(self, nodes, index_struct, storage_context, service_context, show_progress, **kwargs)
...
---> 47     raise ValueError("One of nodes or index_struct must be provided.")
48 if index_struct is not None and nodes is not None:
49     raise ValueError("Only one of nodes or index_struct can be provided.")
ValueError: One of nodes or index_struct must be provided.
Kindly solve the error

Here's how I would do it (assuming you have the latest version installed)
from llama_index import SimpleDirectoryReader, VectorStoreIndex, ServiceContext, set_global_service_context
from llama_index.llms import OpenAI

# setup LLM with max_tokens
llm = OpenAI(model="gpt-3.5-turbo", temperature=0, max_tokens=200)

# setup global service context
ctx = ServiceContext.from_defaults(llm=llm, chunk_size=600)
set_global_service_context(ctx)

# setup index
docs = SimpleDirectoryReader(path).load_data()
index = VectorStoreIndex.from_documents(docs)

# query index, optionally set the top k (default is 2)
query_engine = index.as_query_engine(similarity_top_k=2)
response = query_engine.query("query")

# save the index
index.storage_context.persist(persist_dir="./storage")

# load the index
from llama_index import StorageContext, load_index_from_storage
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./storage"))

Thank you @logan-markewich

import os
from llama_index import SimpleDirectoryReader, VectorStoreIndex, StorageContext, load_index_from_storage
from openai import OpenAI
Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "..................................."
Set the path to the directory where your documents are stored
path = "d:\channel\knowledge"
Setup LLM with max_tokens
llm = OpenAI(model="gpt-3.5-turbo", temperature=0, max_tokens=200)
Setup global service context
ctx = ServiceContext.from_defaults(llm=llm, chunk_size=600)
set_global_service_context(ctx)
Setup index
docs = SimpleDirectoryReader(path).load_data()
index = VectorStoreIndex.from_documents(docs)
Query index, optionally set the top k (default is 2)
query_engine = index.as_query_engine(similarity_top_k=2)
response = query_engine.query("query")
Save the index
index.storage_context.persist(persist_dir="./knowledge")
Load the index
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./knowledge"))

AuthenticationError                       Traceback (most recent call last)
File [c:\Users\Dell\anaconda3\lib\site-packages\tenacity_init_.py:382](file:///C:/Users/Dell/anaconda3/lib/site-packages/tenacity/init.py:382), in Retrying.call(self, fn, *args, **kwargs)
381 try:
--> 382     result = fn(*args, **kwargs)
383 except BaseException:  # noqa: B902
File [c:\Users\Dell\anaconda3\lib\site-packages\llama_index\embeddings\openai.py:165](file:///C:/Users/Dell/anaconda3/lib/site-packages/llama_index/embeddings/openai.py:165), in get_embeddings(list_of_text, engine, **kwargs)
163 list_of_text = [text.replace("\n", " ") for text in list_of_text]
--> 165 data = openai.Embedding.create(input=list_of_text, model=engine, **kwargs).data
166 return [d["embedding"] for d in data]
File [c:\Users\Dell\anaconda3\lib\site-packages\openai\api_resources\embedding.py:33](file:///C:/Users/Dell/anaconda3/lib/site-packages/openai/api_resources/embedding.py:33), in Embedding.create(cls, *args, **kwargs)
32 try:
---> 33     response = super().create(*args, **kwargs)
35     # If a user specifies base64, we'll just return the encoded string.
36     # This is only for the default case.
File [c:\Users\Dell\anaconda3\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py:149](file:///C:/Users/Dell/anaconda3/lib/site-packages/openai/api_resources/abstract/engine_api_resource.py:149), in EngineAPIResource.create(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)
127 @classmethod
128 def create(
129     cls,
(...)
136     **params,
137 ):
...
--> 326     raise retry_exc from fut.exception()
328 if self.wait:
329     sleep = self.wait(retry_state)
RetryError: RetryError[]
why am i getting error if i already mentioned my OPENAI_API_KEY in code ........ here my OPENAI_API_KEY is not showing due to privacy.
kindly solve the error @logan-markewich
thank you

Are you running in a notebook? I've found I have to also set the key directly
import openai
openai.api_key = ...

import openai
openai.api_key = ..
setup LLM with max_tokens
llm = OpenAI(model="gpt-3.5-turbo", temperature=0, max_tokens=200)
setup global service context
ctx = ServiceContext.from_defaults(llm=llm, chunk_size=600)
set_global_service_context(ctx)
setup index
docs = SimpleDirectoryReader(path).load_data()
index = VectorStoreIndex.from_documents(docs)
query index, optionally set the top k (default is 2)
query_engine = index.as_query_engine(similarity_top_k=2)
response = query_engine.query("query")
save the index
index.storage_context.persist(persist_dir="./knowledge")
load the index
from llama_index import StorageContext, load_index_from_storage
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./knowledge"))                                                                                ---------------------------------------------------------------------------
RateLimitError                            Traceback (most recent call last)
File c:\Users\Dell\anaconda3\lib\site-packages\tenacity_init_.py:382, in Retrying.call(self, fn, *args, **kwargs)
381 try:
--> 382     result = fn(*args, **kwargs)
383 except BaseException:  # noqa: B902
File c:\Users\Dell\anaconda3\lib\site-packages\llama_index\embeddings\openai.py:165, in get_embeddings(list_of_text, engine, **kwargs)
163 list_of_text = [text.replace("\n", " ") for text in list_of_text]
--> 165 data = openai.Embedding.create(input=list_of_text, model=engine, **kwargs).data
166 return [d["embedding"] for d in data]
File c:\Users\Dell\anaconda3\lib\site-packages\openai\api_resources\embedding.py:33, in Embedding.create(cls, *args, **kwargs)
32 try:
---> 33     response = super().create(*args, **kwargs)
35     # If a user specifies base64, we'll just return the encoded string.
36     # This is only for the default case.
File c:\Users\Dell\anaconda3\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py:153, in EngineAPIResource.create(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)
138 (
139     deployment_id,
140     engine,
(...)
150     api_key, api_base, api_type, api_version, organization, **params
151 )
...
--> 326     raise retry_exc from fut.exception()
328 if self.wait:
329     sleep = self.wait(retry_state)
RetryError: RetryError[]
solve this error, you can try the following steps:
Check the rate limit for your OpenAI API key. The rate limit specifies the maximum number of requests you can make within a certain time period. Make sure you are not exceeding this limit.                                                                                solve the error
how to know that i am exceeding the limit?
kindly solve the error @logan-markewich
thank you


Propagate main()'s return code in AMPI

AMPI renames the application’s main function to AMPI_Main.
The return value from AMPI_Main is ignored, i.e. the runtime always assumes a return code 0.
Workaround: change return to exit().
We need to propagate the real return code throughout AMPI and TCharm.
Potential issue:
How to handle AMPI_Main without return? E.g. when defined as void main(...), or with implied return 0 at end.
This might then return a random value from the stack.
Edit:

void main() doesn't work with AMPI (even without this PR).
implied return 0 does actually return 0, so seems to be fine.

This is ready for review.


#697: Encode expanded script on Python 2

Fix #697

Hey, @djh82! 👋
I'd really love if you could drop a review here and/or your thoughts — apart from "you shouldn't be wasting time on Python 2.7, Pablo." hahaha 🙈😁 — I promise that's the last one thou.
Thanks in advance!!

Looks reasonable to me! My only question is about the use of replace for the errors argument - there are very few other occasion where that is specified (and they might only be log prints?), the rest of the codebase defaults to strict.
Given we're expecting folk to have typed these commands, we should be expecting valid utf8? I don't see the harm in erroring - I think it would be more surprising to the user to replace characters?

I see your point, @djh82. My thinking was in the case where the user might be in a situation where the encoding issue cannot be solved -- for some reason such as the user is not able to or doesn't know how to.
I've been thinking that thefuck could then issue a warning and do the replacement (or ignore?). That way, it works when something is not right but the user is aware of the problem. What do you think?

I think adding a warning is a reasonable compromise. It's always a compromise, but I aim for the least surprising action for the user.

👍 @djh82, thanks to your inputs, while working a bit more around #697, I got to the same conclusion as you did. If for some reason expanded cannot be encoded, it's better to just let it explode so the user knows there's something wrong with their setup.


ImportError: No module named blessed

Hi, I'm getting this error:
$ lldb a.out
An error occurred while loading Voltron:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/voltron/entry.py", line 43, in <module>
    import blessed
ImportError: No module named blessed

(lldb) target create "a.out"
Current executable set to 'a.out' (x86_64).
(lldb)

I'm using a Homebrew:ed version of Python 3.5. I find it weird -- blessed is installed and should be on the sys.path...?

LLDB will be linked against the system Python, rather than the Homebrew Python, so that's where it will be looking for packages. It's easiest currently just to install Voltron into the system Python's site-packages (or with --user if you like).
I've been meaning to support a better way to do this using a virtualenv, but I haven't done it yet.

On 23 Jun 2016, at 8:50 PM, Tobias<EMAIL_ADDRESS>wrote:
Hi, I'm getting this error:
$ lldb a.out
An error occurred while loading Voltron:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/voltron/entry.py", line 43, in
import blessed
ImportError: No module named blessed
(lldb) target create "a.out"
Current executable set to 'a.out' (x86_64).
(lldb)
I'm using a Homebrew:ed version of Python 3.5. I find it weird -- blessed is installed and should be on the sys.path...?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/snare/voltron/issues/154, or mute the thread https://github.com/notifications/unsubscribe/AAey8URlpyh3ZdIB52ZKKFHVpDok725Gks5qO1P0gaJpZM4I9cVv.

Hey so, what is the solution here? Getting this same error using GDB 7.11.1 on Ubuntu 16.04.1 (LTS)
An error occurred while loading Voltron:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/voltron-0.1.5-py2.7.egg/voltron/entry.py", line 43, in <module>
    import blessed
ImportError: No module named 'blessed'

Should voltron be installed in all dist/site-packages for all Python versions?

Just whichever version of Python GDB is linked against

I guess my actual question is how do I specify the python version to Voltron upon installation?

I wrote some more doco on installing Voltron on systems with multiple Python versions if you guys would like to take a look

Perfect man. Thanks for that. Worked well for me.
On Aug 18, 2016 8:23 AM, "snare"<EMAIL_ADDRESS>wrote:

I wrote some more doco
https://github.com/snare/voltron/wiki/Installation on installing
Voltron on systems with multiple Python versions if you guys would like to
take a look
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/snare/voltron/issues/154#issuecomment-240758409, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAyPjxT8Of9_OeI03i2QkKJbRtTuZG3Pks5qhHj0gaJpZM4I9cVv
.


Remove subscription valid/active/current inconsistencies

So we currently have a few inconsistencies surrounding subscription status/categorization:
Customer
https://github.com/kavdev/dj-stripe/blob/1.0.0/djstripe/models.py#L229
    def _get_valid_subscriptions(self):
        """ Get a list of this customer's valid subscriptions."""
        return [subscription for subscription in self.subscriptions.all() if subscription.is_valid()]

    def has_active_subscription(self, plan=None):
        """
        Checks to see if this customer has an active subscription to the given plan.

        :param plan: The plan for which to check for an active subscription. If plan is None and
                     there exists only one active subscription, this method will check if that subscription
                     is valid. Calling this method with no plan and multiple valid subscriptions for this customer will
                     throw an exception.
        :type plan: Plan or string (plan ID)
        :returns: True if there exists an active subscription, False otherwise.
        :throws: TypeError if ``plan`` is None and more than one active subscription exists for this customer.
        """
        if plan is None:
            valid_subscriptions = self._get_valid_subscriptions()

            if len(valid_subscriptions) == 0:
                return False
            elif len(valid_subscriptions) == 1:
                return True
            else:
                raise TypeError("plan cannot be None if more than one valid subscription exists for this customer.")
        else:
            # Convert Plan to stripe_id
            if isinstance(plan, Plan):
                plan = plan.stripe_id

            return any([subscription.is_valid() for subscription in self.subscriptions.filter(plan__stripe_id=plan)])

    @property
    def active_subscriptions(self):
        """Returns active subscriptions (subscriptions with an active status that end in the future)."""
        return self.subscriptions.filter(
            status=StripeSubscription.STATUS_ACTIVE, current_period_end__gt=timezone.now()
        )

    @property
    def valid_subscriptions(self):
        """Returns this cusotmer's valid subscriptions (subscriptions that aren't cancelled."""
        return self.subscriptions.exclude(status=StripeSubscription.STATUS_CANCELED)

Subscription
https://github.com/kavdev/dj-stripe/blob/1.0.0/djstripe/models.py#L871
    def is_period_current(self):
        """ Returns True if this subscription's period is current, false otherwise."""
        return self.current_period_end > timezone.now() or (self.trial_end and self.trial_end > timezone.now())

    def is_status_current(self):
        """ Returns True if this subscription's status is current (active or trialing), false otherwise."""
        return self.status in ["trialing", "active"]

    def is_status_temporarily_current(self):
        """
        A status is temporarily current when the subscription is canceled with the ``at_period_end`` flag.
        The subscription is still active, but is technically canceled and we're just waiting for it to run out.
        You could use this method to give customers limited service after they've canceled. For example, a video
        on demand service could only allow customers to download their libraries  and do nothing else when their
        subscription is temporarily current.
        """
        return self.canceled_at and self.start < self.canceled_at and self.cancel_at_period_end

    def is_valid(self):
        """ Returns True if this subscription's status and period are current, false otherwise."""
        if not self.is_status_current():
            return False

        if not self.is_period_current():
            return False

        return True

I'd like to get a discussion going on what we want to keep, how we want to define "Valid", "Active", "Current", etc.

Also see #454

From Gitter:

We still have that monkey patch on our (my use) side to treat valid subscriptions to include "past_due". The "past_due" status is still an active subscription in which the charge has failed (and this happens legitimately, I've seen it a few times for us), and "unpaid" is when it has given up trying to charge. The built-in Stripe settings for failed payments includes retrying for a configurable amount of days, then things like "Mark as unpaid" or "Cancel subscription".
So assuming the "past_due" is potentially temporary it seems logically to allow it as valid. Mostly because the user can determine just how long a subscription will remain in "past_due" before it is marked as unpaid or completely cancelled. There may even be a use-case where the library user wants to allow an unpaid subscription to be valid, because they're mental and don't desperately need the money from their customers.

Looks like the second part self.is_period_current may also be wrong here also - As we can see in Stripe's description of The Subscription Cycle, the subscription period elapses when the invoice is created, and is only updated when the payment succeeds. D'oh!  My gut feeling is that we should change this to use the sane defaults from #454 as the values for is_status_current and then remove the is_period_current check by default from is_valid - We could then offer a new setting for users to customise what it means for a subscription to be valid, and use the aforementioned as the default implementation.


Identity Library fail to choose the Data context class and User Class in Add Identity Window

Describe the bug
Identity Library fail to choose the Data context class and User Class in Add Identity Window while add scaffold.
To Reproduce
Steps to reproduce the behavior:

Using this version of ASP.NET Core '2.1'
VS 2017 -> Create a new Asp.Net Core Web Application(MVC)
Add new class below:

ApplicationDbContext
    public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
    {
        public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
            : base(options)
        {
        }
    }

ApplicationUser
    public class ApplicationUser : IdentityUser
    {
    }

Right Click Project->Add Scaffold->Identity->Check Data context class select and User Class

There are no ApplicationDbContext for  Data context class  and ApplicationUser for User Class

Expected behavior
I could choose ApplicationDbContext and ApplicationUser to generate the identity files.
Screenshots

@Eilon can you move this to tooling please?

To report issues with Visual Studio, use the Report a Problem tool.
If you encounter a problem with Visual Studio, we want to know about it so that we can diagnose and fix it. By using the Report a Problem tool, you can collect detailed information about the problem, and send it to Microsoft with just a few button clicks. See here for more details.


Incorrect typing for this.$

In both Closure and Typescript, this.$ should be a map of strings to HTMLElement, but currently return Element. Given that the only way into the this.$ map is to be an element with an id, this probably only includes HTMLElements.

Wait, the source issue is the same 👎


Crate docs build failed for both releases.

Crate name
rimcol
Build failure link
https://docs.rs/crate/rimcol/0.1.1/builds
Additional details
The build completes successfully on the local machine. Not sure why the limit is exceeded as shown in build logs.

@ligmitz
How much memory do you need to build this crate on your machine? We can increase the 3GB limit to a certain extend.
It also looks like the tensorflow doc build itself fails too, with the same error

Thanks @Nemo157 !
I'm closing this issue then, since this has to be solved in tensorflow-sys.
( also, we never enable network access for any crate, see here)

Thanks for the breakdown, I'll try to convey the issue to the tensorflow/rust maintainers and see if they have a potential solution soon.


Why RR-J > MPJPE?

Hi,
Thanks for awesome work! I just have one confusion that I want to clear.
In Table.1 of your paper, you provide quantitative results of evaluations. I am confused why RR-V/RR-J are higher than MPVPE/MPJPE? As per my understanding, it should be like: PA-MPJPE < RR-MPJPE < absolute MPJPE.
Thank you.

Closing this because my question has been answered here


Idea: Add support for Adafruit NeoPixels or other addressable LEDs?

I had an idea for this to use Adafruit NeoPixels (https://learn.adafruit.com/adafruit-neopixel-uberguide/overview) or other addressable LED strips, which contain individually controllable RGB LEDs. This would allow for more flexibility in the way data is displayed.

Well that would be a considerable increase in complexity required for everything, the App, the Sketch, the communication protocol between the two.
Good idea, but I think it's larger than the scope of this project :)


ISPN-7598 Core test suite leaks threads

https://issues.jboss.org/browse/ISPN-7598
https://issues.jboss.org/browse/ISPN-7604

I moved the ISPN-7614 fix to a separate fix, https://github.com/infinispan/infinispan/pull/4985, because unlike the others it mixed important and aesthetic changes in the same commit.

Ups didn't mean to close that one :) But at least now it is fixed hopefully.
On Wed, Mar 15, 2017 at 2:58 PM Dan Berindei<EMAIL_ADDRESS>wrote:

@danberindei commented on this pull request.
In
core/src/test/java/org/infinispan/distexec/ReplSyncDistributedExecutorTest.java
https://github.com/infinispan/infinispan/pull/4972#discussion_r106254868
:

@@ -22,10 +22,4 @@ protected String cacheName() {
protected CacheMode getCacheMode() {
return CacheMode.REPL_SYNC;
}

@Test(groups = "unstable")
@Override
public void testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout() throws Exception {

Well, you https://issues.jboss.org/browse/ISPN-3992 as out of date one
year ago ;)
I'm pretty sure the test was fixed when you changed the return type to
CompletableFuture, I just enabled it to make sure. The failures in the
unstable tests build are irrelevant, because it didn't have the proper
expectedExceptions annotation.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/infinispan/infinispan/pull/4972#discussion_r106254868,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnj73i6ugaom39fA3mgA-8ZA-LyZSBwks5rmDS_gaJpZM4Mczz7
.

Oops, I think this is causing some leaks in the query module :(

The query tests should be fixed now. Turns out the index locks/data caches aren't always internal, and they appear in the set returned by DefaultCacheManager.getCaches(), so I removed the explicit cache stop from TestingUtil.killCacheManagers(), and DefaultCacheManager.stop() should stop them in the right order.

Updated once again to fix ExecutionAuthorizationTest (cache.getCacheManager() requires ADMIN) and ClusteredCacheWithElasticsearchIndexManagerIT (TestingUtil.clearContent() was clearing the data container and the stores, but not the indexes).

@danberindei there are 11 failures

Marking as 9.1, unless there is a bug in actual runtime code here @danberindei ? Just trying to see if we can lighten the CI load temporarily.

@wburns ISPN-7604 is an actual bug. I can open a separate PR for it if you want, but it can wait until the final release I think.

Okay would be easier to wait if it isn't that urgent. Thanks for clarification @danberindei !

PR looks fine to me. Just needs rebase though.

hey @danberindei, there are some unused imports :)
the PR looks good to me.

It makes sense to backport any meaningful fixes to 9.0.x as well

Merged to master, thanks


phpstan showing errors

Laravel Version: 5.4
PHP Version:7.0
Database Driver & Version:MySql

Description:
I am using phpstan to analyze my code. I saw 250 errors while analysing framework folder. May I know why it is happening and is it negligible issues?
Because in my application also has some issues similar to that.
Please help me,
php -d memory_limit=-1 vendor/bin/phpstan analyse vendor/laravel

Don't analyze the framework code, it's fine :)

@themsaid I know framework is working fine :-) but my question why it is showing errors? is it negligible?
How can we configure phpstan to get same quality which framework has?


Add description to egress security group rule

what

Adds a generic description to the egress security group rule

why

It is good practice to have a description
If you use tfsec, rule AWS018 will now pass

No longer needed


EventLoop on main thread?

This looks interesting. Unfortunately it won't run on macOS:
15:16:09 [INFO] added graph a795c3e9-0a2f-48bf-a9e2-03378e8e59b8 : Canvas with 1 versions to internal library
15:16:09 [INFO] added graph fd41d8ef-d10f-4499-8a90-35b73d8ff246 : Log with 1 versions to internal library
15:16:09 [INFO] added internal libraries to catalogue
15:16:09 [INFO] searching in /Users/moritz/code/ProcFlow/data for libraries
15:16:09 [INFO] added graph info /Users/moritz/code/ProcFlow/data/boot/boot: Proc Flow Boot Loader with 1 versions
15:16:09 [INFO] added graph info /Users/moritz/code/ProcFlow/data/graphics/graph_canvas: Proc Flow Graph Canvas with 1 versions
15:16:09 [INFO] added application libraries to catalogue
15:16:09 [INFO] searching in /Users/moritz/Documents/ProcFlow/Libraries for libraries
15:16:09 [WARN] could not walk directory: /Users/moritz/Documents/ProcFlow/Libraries: IO error for operation on /Users/moritz/Documents/ProcFlow/Libraries: No such file or directory (os error 2)
15:16:09 [INFO] added document libraries to catalogue
15:16:09 [INFO] boot sequence initiated for graph a795c3e9-0a2f-48bf-a9e2-03378e8e59b8 version 1
15:16:09 [INFO] graph a795c3e9-0a2f-48bf-a9e2-03378e8e59b8 found in catalogue with name Canvas
15:16:09 [INFO] internal graph a795c3e9-0a2f-48bf-a9e2-03378e8e59b8 : Canvas version 1 node actor spawned
15:16:09 [INFO] engine send value
15:16:09 [INFO] controller send value
15:16:09 [INFO] canvas recv
15:16:09 [INFO] engine value sent
15:16:09 [INFO] canvas edit
thread '<unnamed>' panicked at 'On macOS, `EventLoop` must be created on the main thread!', /../winit-0.22.2/src/platform_impl/macos/event_loop.rs:46:17

Would you consider changing this?

Apologies for not getting back on this. The project runs primarily on Windows _at the moment just due to the nature of running an event loop on a different thread than main. I actually just got done creating an issue on the Amethyst game engines' tracker for allowing such a modification on their project.
The reason for this error is that I am using the Windows winit extension to launch a window when an actor in the axiom actor library gets run. I am actually considering moving the windowing and graphics elements to main thread, but that will take some time. (Ideally I could just keep them there with the node) By moving the window to the main thread where the engine structure itself is run, I can drop the Windows extension from the event loop and it will be cross-platform then.
Finally, this project can run nodes in a graph somewhat decently, but this entire project is in an experimental phase, so don't expect an actual product yet. The goal for this project is to offer an MIT licensed node based procedural editor that is designed to offer some functionality similar in nature to Houdini and Substance Designer. Ideally, I can keep everything library oriented so that the engine can itself be used at runtime within a game or other application as well, hence why I was trying to launch a windowed event loop in the child actor thread.
TLDR; It works in Windows right now and is not cross-platform.

The goal for this project is to offer an MIT licensed node based procedural editor that is designed to offer some functionality similar in nature to Houdini and Substance Designer.

That's exactly what I'm interested in. Basically there is no nice standalone shader editor for OSL and as an ex- professional shader writer writing one in Rust is a pet project in the back of my head.
I also like building little side projects that almost all could benefit, each, from just slapping a node editor on top. :]
I'm a Linux guy but I do most of my work on a small, 2014 MBP these days as I often work from god-knows-where. Hence my interest for getting this to run 'elsewhere'. :)

Understood. I will keep this issue open for now, and close it once I get the window on the main thread. This really is a pet project at the moment, so expect a great deal of delay.


Use Relative Path

List of Changes

Remove some Print statements in basic.py
Made every path to a relative for readability.

Acknowledgement

[x] I have read the Contributing Guidelines

Thanks for catching those prints! Question: how will the absolute -> relative switch change the way we call manim?

Thanks for catching those prints! Question: how will the absolute -> relative switch change the way we call manim?

Nothing but just the way it prints in log statements that's it. In Windows, the tex templates would be mixed with back and front slashes. /,\ for example ./media\Tex\a6d208c81e4e5843.tex and looks weird. That the reason I changed it. :)

I see, that makes sense! I'm still confused... I can see why the media_dir should/would be better as relative, but the library-wide config file shouldn't be changed to relative, right? I mean, this change will affect where the files are found, not only how they are printed. Am I missing something?

Done it!


[release v1.2] Handle committing offsets on revoked partitions

We need to handle the case when the offset tracker was removed from
our Map since when partitions are revoked we remove the associated
offset tracker, however, we may get a response from the sink for a
previously owned partition after a partition has been revoked.
Note: it's not possible to commit offsets of partitions that a
particular consumer instance doesn't own.
Signed-off-by: Pierangelo Di Pilato<EMAIL_ADDRESS>Co-authored-by: Pierangelo Di Pilato<EMAIL_ADDRESS>

/test 49-e2e-aws-ocp-49

/retest

/retest

/test 49-e2e-aws-ocp-49

all green, merging manually this backport...


Add more protection options for RedProtectProtectionModule

Add more protection options for RedProtectProtectionModule

Marking this as stale even though we havent reviewed it yet.
The repo is no merged to the Slimefun org so it should be reviewed faster in the future.


Add more videos

This is just a place holder issue to use github to upload videos XD

https://github.com/ckcr4lyf/EvilAppleJuice-ESP32/assets/6680615/53ed6d7c-0569-4f22-b55b-bc9973c4bc93

https://github.com/ckcr4lyf/EvilAppleJuice-ESP32/assets/6680615/c6e871fd-9fdf-4507-ae21-a566beead5cc


Implement initial message passing capability

This is the start of Scale's new messaging passing architecture to improve scalability.
All of this should be in a Django app, called "messaging"?
Base class CommandMessage

self.succeeded - used by manager to know whether to ack success for this message
self.new_messages - list of CommandMessage, used by manager to know if new messages should be sent
self.type - type of message to use in factory method to create correct subclass, will need to be sent/received as part of overall payload in the CommandMessageManager methods
to_json()
from_json(json_dict)
execute() - docstring should mention that this should be idempotent and able to be run out of order

Subclasses register in Django app methods and have no-arg init()
Factory method to generate CommandMessage objects from JSON payloads
Basic test CommandMessage (log "Hello Scale" or something)
Class CommandMessageManager

init() sets up and initializes actual backend (SQS, RabbitMQ, Kafka) from settings/env vars
send_message(cmd_msg) - uses cmd_msg.to_json() to create payload
singleton access as cmd_msg_mgr
process_messages() - generically handles "loops" of message processing, should handle multiple in a loop (so we can do 10 at a time with SQS batch), gets JSON payloads from backend and calls factory method and from_json() to set up cmd_msg objects, and processes them

Django management command to run cmd_msg_mgr.process_messages()
Classes that abstract/implement backends (SQS, RabbitMQ, Kafka, etc)
SQS backend should use long-polling and SQS batch (10 at a time)
Need to updates docs to discuss configuring Scale to use whatever messaging backend.

So after some reading on the subject, I'm leaning toward RabbitMQ for the out-of-the-box solution. RabbitMQ appears to be the nearest match for what we are trying to accomplish and has excellent Python (and other cross-language support), as a bonus its usage is very similar to SQS.  Kafka isn't a first choice as it is favors an event stream approach no cleanup following message processing (message roll-off happens by queue size / TTL). It is up to the consumer to manage their position in the log instead of just popping off the front of the queue.

The initial implementation will support RabbitMQ, with SQS behind the secondary implementation.


Typo in Cloudformation policy

Spotted a typo in here
https://github.com/Senzing/aws-marketplace-evaluation/blob/960da19cdf007403f565c6ba8262ca1b16737e88/docs/examples/CFn_Policy3.json#L46-L51

PR merged


feat: migrate to cert-manager

This PR...
Changes

Fixes

Checklist

[x] tested locally
[ ] added new dependencies
[ ] updated the docs
[ ] added a test

After working with cert-manager a bit and looking at other options, those are my conclusions and direction I suggest we should go. Starting from cert-manager:
Cert-manager
One of the most popular solutions for cert management in k8s space from what I see. https://cert-manager.io/.
Pros

No configuration needed to deploy basic version. It will work out of the box.
It's quite well-known solution when it comes to cert management in k8s.
If there is already cert-manager deployed in the cluster it's easy to integrate it with monokle-admission-controller (we should enable it through helm charts).
Cluster admin has full control over certificates, their configuration and how those are created.

Cons

Seems like cert-manager is quite complex so it adds complexity to this solution as a whole. It might seem a bit too much for just "managing single certificate for admission webhook server".
I can imagine someone is reluctant to deploy it to their cluster (due to complexity or resources restrains).
Not the easiest to debug if something goes wrong.
Default config is Self-Signed CA (considered not the most secure - however not sure if it matters that much for internal cluster communication 🤔).

How others do it?
Kyverno
I also looked at Kyverno and how they handle this. It has it's own cert-renewer. It uses Self-Signed CAs and also allows providing manually generated certificates.
For default configuration, it's also out of the box one - https://kyverno.io/docs/installation/customization/#default-certificates:

By default, Kyverno will automatically generate self-signed Certificate Authority (CA) and a leaf certificates for use in its webhook registrations. The CA certificate expires after one year. When Kyverno manage its own certificates, it will gracefully handle regeneration upon expiry.

This is done via dedicated custom made renewer util.
The other option is providing own certificat - https://kyverno.io/docs/installation/customization/#custom-certificates:

You can install your own CA-signed certificate, or generate a self-signed CA and use it to sign certificates for admission and cleanup controllers. Once you have a CA and X.509 certificate-key pairs, you can install these as Kubernetes Secrets in your cluster. If Kyverno finds these Secrets, it uses them, otherwise it will fall back on the default certificate management method. When you bring your own certificates, it is your responsibility to manage the regeneration/rotation process. Only RSA is supported for the CA and leaf certificates.

Kong
Did a bit of searching for how other tools handle/advise, on supplying certs for validation controllers. Found Kong (API platform on top of K8s AFAIU). They have a bit similar approach as Kyverno. Either one can provide own cert or, by default, the certificate is generated by install script or helm chart (so there is no built-in cert manager/renewer) - https://docs.konghq.com/kubernetes-ingress-controller/latest/deployment/admission-webhook/#main.

Conclusions
Based on the above I think cert-manager is viable options as it is well-known, allows easy setup and a lot of flexibility. However, we should provide simpler alternative. At this stage, using self signed certificate, generated by the install script during admission installation (e.g. as Kong does, through Helm) would be reasonable approach - it is simple and lightweight.
As for building dedicated, lightweight cert-renewer which will run in-cluster, maybe it can be done in later stages if needed. At this point I think it's not necessary.
WDYT @WitoDelnat @murarustefaan?

To be honest, while I consider cert-manager a very useful approach for securing endpoints in the Kubernetes space, I would refrain from using it in this exact same context as it is a kubernetes plugin and not something you can count on being installed in all clusters.
Now, while installing cert-manager is pretty simple and straigtforward and you can even add it as dependency to a helm-chart, I believe would not make sense just for generating a self-signed certificate as it would require checking whether a existing installation is already in place to not overwrite the CRDs, add at least one permanently running pod in the cluster, creating a Issuer resource configured for self-signed CA's, generating and updating the certificate Resources, etc.
A different approach, depending on how resource-intensive this admission controller pod will be and whether we want it to scale horizontally, would be using an initContainer to generate the CA, public & private keys in:

considering a highly-available deployment, a kubernetes Secret
considering a single-pod deployment, an emptyDir volume.
The initContainer would generate a new certificate when expired or close to expiry, or in the case of using emptyDir, would   generate a new certificate each time (as emptyDir is ephemeral).

Now considering the CA & Certificate generation is complete (even if we would use cert-manager or our own generation method), the controller pod would need to make sure the APIServer has the correct CABundle for the ValidatingWebhookConfiguration, therefore on each bootstrap would need to refresh the ValidatingWebhookConfiguration resource with the valid CA bundle.
@WitoDelnat @olensmar what's you take on this?

@murarustefaan thanks for the suggestion (and explanation). Seems like even more straightforward and maintenance-free approach. I'm taking a look into that 👍

considering a highly-available deployment, a kubernetes Secret

I think this would be the direction. Admission controller should work without any interruptions/downtime, since being unavailable means broken deployment of other (validated) resources (well, depends on failurePolicy config option).

I believe we should refrain from cert-manager in the long run. I thought maybe it was easy to get started with and we could have something temporarily, but its indeed bringing a bulldozer when a hammer would be sufficient. Kyverno has a custom component to deal with certificate renewal.
There are plenty of strategies for generating (or providing custom) certificates. Helm Hook, initContainer, within the NodeJs process, etc. I think we should simply use a Kubernetes Secret as an indirection so this can change over time. It also provides custom certificates out of the box - you'd basically simply need to make sure its there before you bootstrap Monokle Admission Controller.
As for our first strategy, I'm not so opinionated and would go for the simplest solution. initContainers and Helm Hooks are both basically container jobs, I'd maybe lean to a Helm Hook as it works for both single-instance & high available setups.

Yeah, indeed, but using helm hooks would regenerate certificates only during helm upgrades / installs, therefore would require the usage of certificates with large expirations (eg. at least over an year). An initContainer would solve this as pods are pretty ephemeral and even in an ideal cluster where nodes would not get shuffled often I highly doubt any pod would last more than say 3 months.
Furthermore, using helm hooks ties us to only shipping the admission controller setup using Helm, which I would refrein from doing in the long run; that's why i suggested using the bootstrapping of an initContainer as that's more easier to build on top in the future.
However, @f1ames, whichever way you go, pay attention to creating a ServiceAccount, Role & RoleBinding accordingly without granting more privileges than actually needed for reading, creating and updating the CA Secret & the ValidatingWebhookConfiguration resources.


Menu merging broken in 3.1

I recently updated to 3.1.0 and I found that menus in MDI child forms no longer merge in to the MDI parent menu, I reverted to 3.0.6 and the menus merge correctly. I haven't got time to download and look at the source to try to see what's changed - apologies! I'm sticking with 3.0.6 for now as there don't appear to be any issues with my application using that version.

Duplicate to #613 . Please change the docking style.


'Required' validation messages should not be first on the error message order of precedence

Looking at the code, it seems that only one error message is displayed at a time. However, it would be nice that other error messages took precedence over the 'Required' error message. For example, if I have a property of type 'integer' that is required and the user types letters, ideally the user would see a Not a number error message and not a Required error message.
This is currently what I'm seeing:

I have put together a plnkr that shows this.

:+1: This is a big pain point for me as well

How about this, if the viewValue of the form is not empty then the required validation message should not be shown (or last in the list), otherwise it should be first?
I think this pops up since the input is an integer which means that the model value is empty and that's why you get the required instead of the "Not an integer". Marking it as a bug.


Milliseconds support

import schedules, times, os

schedules:
  every(milliseconds=20, id="tick", throttle=30):
    echo("tick ", now())
    sleep(5000)

Fails currently with [DivByZeroDefect]

My bad. Param seconds is still required. Works fine.

@qb-0 Currently, nim-schedules uses Second as the minimal resolution (fromUnix). I guess we can update the library by using fromUnixFloat, which supports subsecond as resolution.

In my experience, the smallest possible interval is 2 seconds. Anything below that will still result in an interval of "every 2 seconds".


Add glossary entries to search results page

If a given search term exists in the Electron glossary for the user's current locale, it should be displayed at the top of the page:
require('./lib/i18n').glossary['en-US'].IPC

{
  term: 'IPC',
  description: 'IPC stands for Inter-Process Communication. Electron uses IPC to send serialized JSON messages between the main and renderer processes.'
}

Kind of like when Google shows a dictionary entry above the search results:

The first step to making this happen would be creating an Algolia search index for the glossary entries: https://github.com/electron/algolia-indices/issues/22


Support for images and relative links in attributed string renderer.

Hi,
I don't know if you are still maintaining this project, but if you are, you might be interested by this PR adding images and relative links to attributed string rendering on both iOS and macOS. Additionally I made a overall refresh to the project: cmark v0.29, Swift 4.2 example apps, paragraph attributes…
(see the commit message for all details).

Wow, this is great! Thanks for working on this — I'll try to review as soon as possible.

Any update on whether this is going to be merged in or not? We are looking at replacing our current Markdown library, and image functionality is crucial.  Thanks!

Sorry about the delay in getting to this, this looks great, merging!


Not every prefix is used on ethernet

when a /27 prefix is created only 30 free IPs are mentionned. Although this is valid for Ethernet segments it is confusing when the prefix is used as a pool for nat addresses or aggregating loopbacks. It might be usefull to add a flag to a prefix if it is an ethernet segement or just a container for something else

It's pretty easy to modify this behavior, but we need to figure out how to indicate whether the network/broadcast IPs should be counted. It might make sense to add a new prefix type called "pool" and count otherwise unusable IPs for prefixes of this type. It might also make sense to count them for container prefixes.

FWIW, RackTables work around this by using "reserved" addresses, that a checkbox permits to allocate easily while creating the prefix. That way, every prefix is useable as a pool or not.


Cannot run bugsnag when using app.config.js (dynamic app.json)

Describe the bug
Modern expos allow using app.config.js rather than app.json.
If app.json is missing, or is missing an "expo": {} element, the init command fails:
zumthor$ npx bugsnag-expo-cli init
✔ @bugsnag/expo already appears to be installed, do you want to install it and its dependencies anyway? … no
✔ What is your Bugsnag API key? … deadbeef4acab110b32b3a1234567890
> Inserting API key into app.json

  TypeError: Cannot read properties of undefined (reading 'extra')
      at module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/lib/set-api-key.js:9:33)
      at async module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/commands/set-api-key.js:18:15)
      at async module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/commands/init.js:11:3)
      at async go (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/cli.js:39:21)

Workaround

Create a placeholder app.json: echo '{ "expo": {} }' > app.json
Follow the instructions as given
hand-replay the new parts of app.json into the config
remove app.json

Steps to reproduce

switch from app.json to app.config.js
run npx bugsnag-exp-cli init
dies editing app.json

It will also die if the app.json lacks an expo field: i.e. app.json == {} yields sadness, app.json = { "expo": {} } yields workaround.
Example code snippet
Vanilla repo, change to dynamic config:
cd /tmp
expo init foo -t blank
cd foo
# (will attest that the config hasn't changed)
npx expo config --json | jq '. | del(._internal) | { expo: . }' > cfg-vanilla.json
( echo -n "const AppConfig = (" ; cat app.json ; echo ")" ; echo  "export default AppConfig" ) > app.config.js
mv app.json away-app.json
npx expo config --json | jq '. | del(._internal) | { expo: . }' > cfg-dynamic.json

# this will be empty, the configs are the same
diff cfg-dynamic.json cfg-vanilla.json

Demonstrate the bug:
quickbeam main /tmp/foo$ npx bugsnag-expo-cli init

✔ @bugsnag/expo does not appear to be installed, do you want to install it and its dependencies? … yes
✔ If you want to install @bugsnag/expo ^46.0.0 hit enter, otherwise type the version you want … ^46.0.0
> Installing @bugsnag/expo. This could take a while!
✔ What is your Bugsnag API key? … deadbeef4acab110b32b3a1234567890
> Inserting API key into app.json

  Error: Couldn’t find app.json in "/private/tmp/foo". ....

Demonstrate the workaround:
/tmp/foo$ echo '{ "expo": {} }' > app.json
/tmp/foo$ npx bugsnag-expo-cli init
/tmp/foo$ cat app.json

... then replay the changes from app.json into your config file.
Environment

Bugsnag version: 46
Expo SDK version (e.g. SDK 44): 46
Device (e.g. iphonex): cli

Error messages:

  TypeError: Cannot read properties of undefined (reading 'extra')
      at module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/lib/set-api-key.js:9:33)
      at async module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/commands/set-api-key.js:18:15)
      at async module.exports (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/commands/init.js:11:3)
      at async go (/Users/flip/.npm/_npx/c6a5a58d15e1cf59/node_modules/bugsnag-expo-cli/cli.js:39:21)

Hi @mrflip - thanks for raising this. We're going to investigate and will keep you updated on this thread 👍

Hi @mrflip - we just released https://github.com/bugsnag/bugsnag-expo/releases/tag/v46.0.2 of bugsnag-expo, which includes a fix for automated installations where apps use app.config.js 🎉
Thanks again for reporting this!


Federation not working

Using types from other service just does not work. I can't define external derectives properly, and get errors about must being used with @requires, @provides, or as a key type. I cannot get it to work. Let me link you to the thread of discussion
https://spectrum.chat/apollo/apollo-federation/federation-query-inside-mutation~3c9cca0e-9a82-4738-8f9f-5d9de0c3876d
and the discussion on graphile discord which resulted in no solution:
I have a question about how federation is supposed to work. I have postgraphile setup and working and can see my schema. Now in another service i want to have a mutation that returns one of the types made from the postgraphile service. Let's call this type 'User' and the user has some nested fields one of which is 'UUID'. When i try to reference this type apollo complains about unknown type. The only way i can fix this is if i extend the type and basically redefine every field and add @key and @external directives. This seems pointless to use federation if i have to redefine the types i want to share in every new service?? What am i misunderstanding?

extend type User @key(fields: "nodeId") {
    id: UUID! @external
    name: String! @external
    position: String! @external}

this is the type i want to share. I extend it like that in the service i want to use it from, but it's made by postgraphile itself
i'm trying to use it like this

type Mutation { doaThing(blah: String) : User @external }

solution documented at end of spectrum post. Needed a resolver with __typename and to introduce an intermediate type.

Thanks for sharing


Add RVSDG can-subst-beneath

Summary
This PR adds can-subst-*-beneath, which allows expressions/bodies/operands to be substituted for another, as described in this doc.
More thorough examples are included in the doc and in the tests, but briefly:

The key relations are (relation can-subst-TYPE-beneath (Body TYPE TYPE)), where TYPE is one of {Expr, Body, Operand, VecOperand, VecVecOperand}.
Add (can-subst-TYPE-beneath above from to) if it is sound to replace zero or more occurrences of from with to, strictly within above.
Then, saturate the subst ruleset for the appropriate unions to be made.

Testing
Added tests to src/rvsdg/tests.rs

Looks like you are also getting the weird UnicodeNotAllowed error in CI, I have no idea what's causing it.


python312Packages.pepit: init at 0.3.2

Description of changes
PEPit: Performance Estimation in Python
https://pepit.readthedocs.io/
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[x] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 295637

aarch64-darwin

:white_check_mark: 4 packages built:

python311Packages.pepit
python311Packages.pepit.dist
python312Packages.pepit
python312Packages.pepit.dist


call: no common audio codecs - rejected

I'm testing rtpengine's transcoding capabilities, but have a problem with baresip not accepting sdp from it.
I have configure baresip account so that it only support opus/48000/2 codec and /uastat shows:
Account:
...
 audio_codecs: opus/48000/2

When INVITE comes in, it has this kind of sdp:
.
v=0.
o=- 686726419<PHONE_NUMBER> IN IP4 <IP_ADDRESS>.
s=-.
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 30244 RTP/AVP 8 0 101 96.
c=IN IP4 <IP_ADDRESS>.
a=label:1.
a=rtcp-rsize.
a=ssrc:3498303314<EMAIL_ADDRESS>a=rtpmap:8 PCMA/8000.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=rtpmap:96 opus/48000/2.
a=fmtp:101 0-15.
...

Dots at the end of lines are added by ngrep.
When baresip receices the INVITE, it respods with 488 Not Acceptable Here and writes to console:
call: no common audio codecs - rejected

Can anyone spot what is wrong with the SDP or is this a bug in baresip?

I made a test where I removed 101 from the SDP before INVITE arrived to baresip:
v=0.
o=- 335166208<PHONE_NUMBER> IN IP4 <IP_ADDRESS>.
s=-.
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 30000 RTP/AVP 96.
c=IN IP4 <IP_ADDRESS>.
a=label:1.
a=rtcp-rsize.
a=ssrc:3776330263<EMAIL_ADDRESS>a=rtpmap:96 opus/48000/2.

and then baresip accepted the INVITE.
So either there is a bug in baresip (it assumes that codec IDs need are in increasing order on m line) or there is an RFC somewhere that tells that they have to be in increasing order.
I'll add bug label to this until such an RFC is found.

I checked https://tools.ietf.org/html/rfc4566#section-5.14 and didn't find anything about ordering of codec IDs on m line.

I tested with pjsua and it accepted INVITE with m-line where opus 96 is after 101:
m=audio 31174 RTP/AVP 101 96
c=IN IP4 <IP_ADDRESS>
a=label:1
a=rtcp-rsize
a=ssrc:1006889197<EMAIL_ADDRESS>a=rtpmap:101 telephone-event/8000
a=rtpmap:96 opus/48000/2
a=fmtp:101 0-15

When I replace pjsua UAS with baresip, exactly the same call fails with
call: no common audio codecs - rejected.
I then tried to figure out from baresip code, where the bug is.  Check of common codec is made by function have_common_audio_codecs  on line:
sc = sdp_media_rformat(stream_sdpmedia(audio_strm(call->audio)), NULL);

and those function calls were not easy to follow.

here is the SDP debug for your session:
SDP session
  local attributes:
    tool='baresip 0.6.2'
  remote attributes:
session media:
audio RTP/AVP
  local formats:
    101 telephone-event/8000/1 (0-15) *
     96 opus/48000/2 (stereo=1;sprop-stereo=1;maxaveragebitrate=28000) *
  remote formats:
      8 PCMA/8000/1
      0 PCMU/8000/1
    101 telephone-event/8000/1 (0-15) *
     96 opus/48000/2 *
  local attributes:
    label='1'
    rtcp-rsize
    ssrc='4029404324<EMAIL_ADDRESS>    ptime='20'
  remote attributes:
local media:

and this is part of your SDP:
a=rtpmap:8 PCMA/8000.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=rtpmap:96 opus/48000/2.

the codec id can be in any order, but the codecs are listed in the order of priority.
so what it says is that telephone-event has a higher priority than opus.
here is the code that decides for a common codec:
static bool have_common_audio_codecs(const struct call *call)
{
	const struct sdp_format *sc;
	struct aucodec *ac;

	sc = sdp_media_rformat(stream_sdpmedia(audio_strm(call->audio)), NULL);
	if (!sc)
		return false;

	ac = sc->data;  /* note: this will exclude telephone-event */

	return ac != NULL;
}

the sdp_format's sc->data points to a valid aucodec for all audio codecs,
except for telephone-event which uses sc->data == NULL
so in your case the function sdp_media_rformat will return telephone-event
and then return false because ac is NULL.
I have analyzed hundreds of SDP messages, and in most cases the telephone-event
always is listed as the last codec.
I think the SDP from RtpEngine is a bit strange, why would it list opus as the last codec ?

Alfred E. Heggestad writes:

I have analyzed hundreds of SDP messages, and in most cases the
telephone-event always is listed as the last codec.
I think the SDP from RtpEngine is a bit strange, why would it list
opus as the last codec ?

When rtpengine is given instructions to transcode audio to a given codec
(in the example with flag codec-transcode=opus), it appends the target
codec as the last codec on m=audio line.
Would it make sense to pick the common non-telephone-event audio codec
no matter which position it has on the list?

okay, could you please try this patch and see if it works better:
diff --git a/src/call.c b/src/call.c
index f98b3ce..2e77bbf 100644
--- a/src/call.c
+++ b/src/call.c
@@ -1430,12 +1430,39 @@ static void sipsess_close_handler(int err, const struct sip_msg *msg,
 }

+static const struct sdp_format *sdp_media_rcodec(const struct sdp_media *m)
+{
+	const struct list *lst;
+	struct le *le;
+
+	if (!m || !sdp_media_rport(m))
+		return NULL;
+
+	lst = sdp_media_format_lst(m, false);
+
+	for (le=list_head(lst); le; le=le->next) {
+
+		const struct sdp_format *fmt = le->data;
+
+		if (!fmt->sup)
+			continue;
+
+		if (!fmt->data)
+			continue;
+
+		return fmt;
+	}
+
+	return NULL;
+}
+
+
 static bool have_common_audio_codecs(const struct call *call)
 {
 	const struct sdp_format *sc;
 	struct aucodec *ac;

-	sc = sdp_media_rformat(stream_sdpmedia(audio_strm(call->audio)), NULL);
+	sc = sdp_media_rcodec(stream_sdpmedia(audio_strm(call->audio)));
 	if (!sc)
 		return false;

juha-h writes:

Would it make sense to pick the common non-telephone-event audio codec
no matter which position it has on the list?

So the algorithm would be:

go through list of account's audio_codecs in order until match is
found in incoming sdp

if incoming sdp has telephone-event audio codec, include also that in
the response

Alfred E. Heggestad writes:

okay, could you please try this patch and see if it works better:

I tried and now baresip accepted invite sdp:
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 30538 RTP/AVP 96 8 101 0.
c=IN IP4 <IP_ADDRESS>.
a=label:1.
a=rtcp-rsize.
a=ssrc:94060335<EMAIL_ADDRESS>a=rtpmap:96 opus/48000/2.
a=rtpmap:8 PCMA/8000.
a=rtpmap:101 telephone-event/8000.
a=rtpmap:0 PCMU/8000.
a=sendrecv.

with response sdp:
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 24786 RTP/AVP 101 0.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-15.
a=rtpmap:0 PCMU/8000.
a=sendrecv.

But the problem now is that baresip is not sending any audio out.
Hangup shows:
audio           Transmit:     Receive:
packets:              0          317
avg. bitrate:       0.0         61.7  (kbit/s)
errors:               0            0
pkt.report:           0          247
lost:                 0            0
jitter:             0.0          7.8  (ms)

IP <IP_ADDRESS> is pingable from baresip host and I have not configured
any sip_listen address.
If I replace baresip with pjsua, audio flows both ways.

thanks, it was just a simple test for now.
we need to replace all calls to sdp_media_rformat with the new function.
(for audio only).
if you prefer to have this behaviour, I can go ahead and create a proper patch/branch.

Alfred E. Heggestad writes:

thanks, it was just a simple test for now.

we need to replace all calls to sdp_media_rformat with the new function.
(for audio only).
if you prefer to have this behaviour, I can go ahead and create a
proper patch/branch.

How about audio transmission, how to get that activated?

I rewrote the patch in a better way:
https://github.com/alfredh/baresip/tree/sdp_remote_codec
the logic is the same as before, except that it will exclude telephone-event codecs.
it searches the list of remote codecs from the top, and selects the first one that
is also supported locally.

Alfred E. Heggestad writes:

I rewrote the patch in a better way:
https://github.com/alfredh/baresip/tree/sdp_remote_codec

Thanks, will try tomorrow morning.

Alfred E. Heggestad writes:

I rewrote the patch in a better way:
https://github.com/alfredh/baresip/tree/sdp_remote_codec

This worked very well.  Now baresip accepted the INVITE and also
transmitted audio after responding with 200 OK.
Below is an example, where UAS baresip has ;audio_codecs=PCMU.
INVITE sip:test-0x55ed3a81a030@<IP_ADDRESS>:5066;transport=tcp SIP/2.0.
Record-Route: <sip:<IP_ADDRESS>;transport=tcp;lr;pm;tcf=PCMA;tct=PCMU>.
Via: SIP/2.0/TCP <IP_ADDRESS>;branch=z9hG4bK0d66.32e432160ecd30af23e800218c1fa0a0.0;i=2.
Via: SIP/2.0/TCP <IP_ADDRESS>:42219;received=<IP_ADDRESS>;branch=z9hG4bK3a20ca12d0e0dbf3;rport=33974.
Contact: <sip:jh@test.tutpro.com;gr=urn:uuid:55860551-f714-931b-2ad9-9ca935948d8b>.
Max-Forwards: 69.
To<EMAIL_ADDRESS>From: <sip:jh@test.tutpro.com>;tag=8eb151bc8c12f1f7.
Call-ID: 379775fd980216b4.
CSeq: 17673 INVITE.
User-Agent: baresip v0.6.2 (x86_64/linux).
Allow: INVITE,ACK,BYE,CANCEL,OPTIONS,NOTIFY,SUBSCRIBE,INFO,MESSAGE,REFER.
Supported: gruu,path,outbound.
Content-Type: application/sdp.
Content-Length: 669.
P-Asserted-Identity<EMAIL_ADDRESS>.
v=0.
o=-<PHONE_NUMBER> 325357272 IN IP4 <IP_ADDRESS>.
s=-.
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 30610 RTP/AVP 96 8 101 0.
a=label:1.
a=rtcp-rsize.
a=ssrc:3797424177<EMAIL_ADDRESS>a=rtpmap:96 opus/48000/2.
a=rtpmap:8 PCMA/8000.
a=rtpmap:101 telephone-event/8000.
a=rtpmap:0 PCMU/8000.
a=fmtp:96 stereo=1;sprop-stereo=1;maxaveragebitrate=28000;cbr=0;useinbandfec=1.
a=fmtp:101 0-15.
a=sendrecv.
a=rtcp:30611.
a=ptime:20.
a=ice-ufrag:SBCRyX2D.
a=ice-pwd:oKKRaXDA4Qga7TzWFr9TQwwB72.
a=candidate:vKBm2HKdJjQPXPvN 1 UDP<PHONE_NUMBER> <IP_ADDRESS> 30610 typ host.
a=candidate:vKBm2HKdJjQPXPvN 2 UDP<PHONE_NUMBER> <IP_ADDRESS> 30611 typ host.

SIP/2.0 200 Answering.
Record-Route: <sip:<IP_ADDRESS>;transport=tcp;lr;pm;tcf=PCMA;tct=PCMU>.
Via: SIP/2.0/TCP <IP_ADDRESS>;branch=z9hG4bK0d66.32e432160ecd30af23e800218c1fa0a0.0;i=2.
Via: SIP/2.0/TCP <IP_ADDRESS>:42219;received=<IP_ADDRESS>;branch=z9hG4bK3a20ca12d0e0dbf3;rport=33974.
To: <sip:test@test.tutpro.com>;tag=ea28446111cec08f.
From: <sip:jh@test.tutpro.com>;tag=8eb151bc8c12f1f7.
Call-ID: 379775fd980216b4.
CSeq: 17673 INVITE.
Server: baresip v0.6.2 (x86_64/linux).
Contact: <sip:test@test.tutpro.com;gr=urn:uuid:9b21c1a8-dc58-a3d6-83d5-7be67cbe70ce>.
Allow: INVITE,ACK,BYE,CANCEL,OPTIONS,NOTIFY,SUBSCRIBE,INFO,MESSAGE,REFER.
Content-Type: application/sdp.
Content-Length: 372.
.
v=0.
o=-<PHONE_NUMBER><PHONE_NUMBER> IN IP4 <IP_ADDRESS>.
s=-.
c=IN IP4 <IP_ADDRESS>.
t=0 0.
a=tool:baresip 0.6.2.
m=audio 9966 RTP/AVP 101 0.
c=IN IP4 <IP_ADDRESS>.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-15.
a=rtpmap:0 PCMU/8000.
a=rtcp:9967 IN IP4 <IP_ADDRESS>.
a=sendrecv.
a=label:1.
a=rtcp-rsize.
a=ssrc:4039866832<EMAIL_ADDRESS>a=ptime:20.

thanks for reporting and testing.
should I merge that branch to master?

Alfred E. Heggestad writes:

should I merge that branch to master?

Yes, please do.


Create ShadowView.py

It is dangerous to have a module called ui and a file called UI.py.  Python can keep these things separate but programmers may get confused.  Once ShadowView is working properly, you should be able to remove UI.py and UI.pyui.  Some modifications also need to be made to Popup.pyui. I can submit those changes once ShadowView.py & .pyui are in the repo.
Class names like ShadowView should be in CamelCase to help programmers quickly spot that they are indeed classes.
The variable name i should be reserved for integer values and not for Images or ImageViews.

The UI.py file is not only about the ShadowView, and the UI.pyui file has nothing to do with the ShadowView, it's the main view that allows you to choose an image or take a picture. I think UI.pyui is easy enough to convert to code, as it's really simple. Maybe a better file structure would be:
main.py - run this script
ShadowView.py - contains only the ShadowView class
Image2ASCII.py - contains conversion code (unchanged)
Popup.pyui - does what it did before (unchanged)
UI.pyui is codified and put into main.py
main.py imports ShadowView, and Image2ASCII.
What do you think of this approach

I reworked the file structure.


Fix error messages for string responses

While match_response_schema  can now accept a string (thanks to https://github.com/thoughtbot/json_matchers/pull/43), I noticed it will cause a NoMethodError in error messages:
NoMethodError:
  undefined method `body' for "{\"foo\":\"bar\"}":String

This fix checks if response respond_to?(:body) before attempting to call it from the error message. Included a test for failed string responses to ensure desired behavior.
Also fixed what I think is a related typo in README: match_json_schema should be match_response_schema, correct?

Released in https://github.com/thoughtbot/json_matchers/releases/tag/v0.6.3.
Thanks!


Travis

Complementary to #32.  This is a very simple Travis config but should help assess new PRs.

:+1: Good idea.

@jswhit , if this PR is merged before #32, we can see if indeed Travis picks up the problem, then hopefully merge #32 to fix it!
And if that all works, perhaps we could then get a new release so that continuum can build on windows (
https://github.com/ioos/conda-recipes/issues/534#issuecomment-157354393)?

@rsignell-usgs This PR brings #32 with it.  I kind of wanted to see a green Travis-CI :wink:
@jswhit Someone with write access needs to activated Travis-CI for the repository.

travis is enabled, merging now...


[SPARK-33408][SPARK-32354][K8S][R] Use R 3.6.3 in K8s R image and re-enable RTestsSuite

What changes were proposed in this pull request?
This PR aims to use R 3.6.3 in K8s R image and re-enable RTestsSuite.
Why are the changes needed?
Jenkins Server is using R 3.6.3.
+ SPARK_HOME=/home/jenkins/workspace/SparkPullRequestBuilder-K8s
+ /usr/bin/R CMD check --as-cran --no-tests SparkR_3.1.0.tar.gz
* using log directory ‘/home/jenkins/workspace/SparkPullRequestBuilder-K8s/R/SparkR.Rcheck’
* using R version 3.6.3 (2020-02-29)

OpenJDK docker image is using R 3.5.2 (2018-12-20) which is old and currently spark-3.0.1 fails to run SparkR.
$ cd spark-3.0.1-bin-hadoop3.2

$ bin/docker-image-tool.sh -R kubernetes/dockerfiles/spark/bindings/R/Dockerfile -n build
...
	 exit code: 1
	 termination reason: Error
...

$ bin/spark-submit --master k8s://https://<IP_ADDRESS>:8443 --deploy-mode cluster --conf spark.kubernetes.container.image=spark-r:latest local:///opt/spark/examples/src/main/r/dataframe.R

$ k logs dataframe-r-b1c14b75b0c09eeb-driver
...
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=<IP_ADDRESS> --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.RRunner local:///opt/spark/examples/src/main/r/dataframe.R
20/11/10 06:03:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error: package or namespace load failed for ‘SparkR’ in rbind(info, getNamespaceInfo(env, "S3methods")):
 number of columns of matrices must match (see arg 2)
In addition: Warning message:
package ‘SparkR’ was built under R version 4.0.2
Execution halted

In addition, this PR aims to recover the test coverage.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass K8S IT Jenkins job.

Test build #130142 has started for PR 30130 at commit 9c2d522.

Test build #130142 has finished for PR 30130 at commit 9c2d522.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130142/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34749/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34749/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34749/
Test FAILed.

Test build #130249 has started for PR 30130 at commit 01666ef.

Test build #130249 has finished for PR 30130 at commit 01666ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130249/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34849/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34849/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34849/
Test FAILed.

Retest this please

Test build #130299 has started for PR 30130 at commit 01666ef.

Test build #130299 has finished for PR 30130 at commit 01666ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130299/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34900/

Test build #130303 has started for PR 30130 at commit 2215566.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34900/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34900/
Test FAILed.

Test build #130303 has finished for PR 30130 at commit 2215566.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130303/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34905/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34905/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34905/
Test FAILed.

Test build #130827 has started for PR 30130 at commit b469663.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35436/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35436/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35436/
Test FAILed.

Test build #130827 has finished for PR 30130 at commit b469663.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130827/
Test FAILed.

Test build #130838 has started for PR 30130 at commit 9f68b8c.

Test build #130838 has finished for PR 30130 at commit 9f68b8c.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130838/
Test FAILed.

Merged build finished. Test FAILed.

Test build #130839 has started for PR 30130 at commit fef0f36.

Thank you, @HyukjinKwon .

Test build #130839 has finished for PR 30130 at commit fef0f36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130839/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35447/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35447/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35447/
Test FAILed.

Hmm. It seems that there is another reason. I'll take a look again.

Retest this please.

Test build #130846 has started for PR 30130 at commit fef0f36.

Test build #130846 has finished for PR 30130 at commit fef0f36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130846/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35448/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35448/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35448/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35454/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35454/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35454/
Test FAILed.

Test build #130964 has started for PR 30130 at commit 1dc2266.

Test build #130964 has finished for PR 30130 at commit 1dc2266.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130964/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35570/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35570/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35570/
Test PASSed.

Hi, @HyukjinKwon
Could you review this again?

Hi, @viirya . Could you review this? This recovers K8s R integration test for Apache Spark 3.1.

Merged to master.

Thank you, @HyukjinKwon and @viirya !


[requires.io] dependency update on master branch

Codecov Report

Merging #67 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #67   +/-   ##
=======================================
  Coverage   61.25%   61.25%
=======================================
  Files          69       69
  Lines        2075     2075
  Branches      162      162
=======================================
  Hits         1271     1271
  Misses        767      767
  Partials       37       37

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update efa90b1...6af9fd9. Read the comment docs.